vllm/docs/README.md

hide
navigation toc

Welcome to vLLM

{ align="center" alt="vLLM Light" class="logo-light" width="60%" } { align="center" alt="vLLM Dark" class="logo-dark" width="60%" }

Easy, fast, and cheap LLM serving for everyone

Star Watch Fork

vLLM is a fast and easy-to-use library for LLM inference and serving.

Originally developed in the Sky Computing Lab at UC Berkeley, vLLM has evolved into a community-driven project with contributions from both academia and industry.

Where to get started with vLLM depends on the type of user. If you are looking to:

• Run open-source models on vLLM, we recommend starting with the Quickstart Guide
• Build applications with vLLM, we recommend starting with the User Guide
• Build vLLM, we recommend starting with Developer Guide

For information about the development of vLLM, see:

• Roadmap
• Releases

vLLM is fast with:

• State-of-the-art serving throughput
• Efficient management of attention key and value memory with PagedAttention
• Continuous batching of incoming requests
• Fast model execution with CUDA/HIP graph
• Quantization: GPTQ, AWQ, INT4, INT8, and FP8
• Optimized CUDA kernels, including integration with FlashAttention and FlashInfer.
• Speculative decoding
• Chunked prefill

vLLM is flexible and easy to use with:

• Seamless integration with popular HuggingFace models
• High-throughput serving with various decoding algorithms, including parallel sampling, beam search, and more
• Tensor, pipeline, data and expert parallelism support for distributed inference
• Streaming outputs
• OpenAI-compatible API server
• Support for NVIDIA GPUs, AMD CPUs and GPUs, Intel CPUs and GPUs, PowerPC CPUs, and TPU. Additionally, support for diverse hardware plugins such as Intel Gaudi, IBM Spyre and Huawei Ascend.
• Prefix caching support
• Multi-LoRA support

For more information, check out the following:

• vLLM announcing blog post (intro to PagedAttention)
• vLLM paper (SOSP 2023)
• How continuous batching enables 23x throughput in LLM inference while reducing p50 latency by Cade Daniel et al.
• vLLM Meetups