mirror of
https://git.datalinker.icu/vllm-project/vllm.git
synced 2025-12-10 16:15:40 +08:00
8d32dc603d
Signed-off-by: xinyuxiao <xinyuxiao2024@gmail.com> Co-authored-by: xinyuxiao <xinyuxiao2024@gmail.com>
322 B
322 B
(quantization-index)=
Quantization
Quantization trades off model precision for smaller memory footprint, allowing large models to be run on a wider range of devices.
:::{toctree} :caption: Contents :maxdepth: 1
supported_hardware auto_awq bnb bitblas gguf gptqmodel int4 int8 fp8 quark quantized_kvcache torchao :::