vllm/docs/features/quantization/README.md

Quantization

Quantization trades off model precision for smaller memory footprint, allowing large models to be run on a wider range of devices.

Contents:

• AutoAWQ
• AutoRound
• BitsAndBytes
• BitBLAS
• GGUF
• GPTQModel
• INC
• INT4 W4A16
• INT8 W8A8
• FP8 W8A8
• NVIDIA TensorRT Model Optimizer
• AMD Quark
• Quantized KV Cache
• TorchAO

Supported Hardware

The table below shows the compatibility of various quantization implementations with different hardware platforms in vLLM:

Implementation	Volta	Turing	Ampere	Ada	Hopper	AMD GPU	Intel GPU	Intel Gaudi	x86 CPU
AWQ	❌	✅︎	✅︎	✅︎	✅︎	❌	✅︎	❌	✅︎
GPTQ	✅︎	✅︎	✅︎	✅︎	✅︎	❌	✅︎	❌	✅︎
Marlin (GPTQ/AWQ/FP8)	❌	❌	✅︎	✅︎	✅︎	❌	❌	❌	❌
INT8 (W8A8)	❌	✅︎	✅︎	✅︎	✅︎	❌	❌	❌	✅︎
FP8 (W8A8)	❌	❌	❌	✅︎	✅︎	✅︎	❌	❌	❌
BitBLAS	✅︎	✅	✅︎	✅︎	✅︎	❌	❌	❌	❌
BitBLAS (GPTQ)	❌	❌	✅︎	✅︎	✅︎	❌	❌	❌	❌
bitsandbytes	✅︎	✅︎	✅︎	✅︎	✅︎	❌	❌	❌	❌
DeepSpeedFP	✅︎	✅︎	✅︎	✅︎	✅︎	❌	❌	❌	❌
GGUF	✅︎	✅︎	✅︎	✅︎	✅︎	✅︎	❌	❌	❌
INC (W8A8)	❌	❌	❌	❌	❌	❌	❌	✅︎	❌

• Volta refers to SM 7.0, Turing to SM 7.5, Ampere to SM 8.0/8.6, Ada to SM 8.9, and Hopper to SM 9.0.
• ✅︎ indicates that the quantization method is supported on the specified hardware.
• ❌ indicates that the quantization method is not supported on the specified hardware.

!!! note For information on quantization support on Google TPU, please refer to the TPU-Inference Recommended Models and Features documentation.

!!! note This compatibility chart is subject to change as vLLM continues to evolve and expand its support for different hardware platforms and quantization methods.

For the most up-to-date information on hardware support and quantization methods, please refer to [vllm/model_executor/layers/quantization](../../../vllm/model_executor/layers/quantization) or consult with the vLLM development team.