[doc] Fold long code blocks to improve readability (#19926)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

docs/ci/update_pytorch_version.md

@@ -91,7 +91,7 @@ source to unblock the update process.
 ### FlashInfer
 Here is how to build and install it from source with torch2.7.0+cu128 in vLLM [Dockerfile](https://github.com/vllm-project/vllm/blob/27bebcd89792d5c4b08af7a65095759526f2f9e1/docker/Dockerfile#L259-L271):
 
-```
+```bash
 export TORCH_CUDA_ARCH_LIST='7.5 8.0 8.9 9.0 10.0+PTX'
 export FLASHINFER_ENABLE_SM90=1
 uv pip install --system --no-build-isolation "git+https://github.com/flashinfer-ai/flashinfer@v0.2.6.post1"
@@ -105,14 +105,14 @@ team if you want to get the package published there.
 ### xFormers
 Similar to FlashInfer, here is how to build and install xFormers from source:
 
-```
+```bash
 export TORCH_CUDA_ARCH_LIST='7.0 7.5 8.0 8.9 9.0 10.0+PTX'
 MAX_JOBS=16 uv pip install --system --no-build-isolation "git+https://github.com/facebookresearch/xformers@v0.0.30"
 ```
 
 ### Mamba
 
-```
+```bash
 uv pip install --system --no-build-isolation "git+https://github.com/state-spaces/mamba@v2.2.4"
 ```
 

docs/cli/README.md

@@ -16,35 +16,33 @@ vllm {chat,complete,serve,bench,collect-env,run-batch}
 
 Start the vLLM OpenAI Compatible API server.
 
-Examples:
+??? Examples
 
-```bash
+    ```bash
-# Start with a model
+    # Start with a model
-vllm serve meta-llama/Llama-2-7b-hf
+    vllm serve meta-llama/Llama-2-7b-hf
 
-# Specify the port
+    # Specify the port
-vllm serve meta-llama/Llama-2-7b-hf --port 8100
+    vllm serve meta-llama/Llama-2-7b-hf --port 8100
 
-# Check with --help for more options
+    # Check with --help for more options
-# To list all groups
+    # To list all groups
-vllm serve --help=listgroup
+    vllm serve --help=listgroup
 
-# To view a argument group
+    # To view a argument group
-vllm serve --help=ModelConfig
+    vllm serve --help=ModelConfig
 
-# To view a single argument
+    # To view a single argument
-vllm serve --help=max-num-seqs
+    vllm serve --help=max-num-seqs
 
-# To search by keyword
+    # To search by keyword
-vllm serve --help=max
+    vllm serve --help=max
-```
+    ```
 
 ## chat
 
 Generate chat completions via the running API server.
 
-Examples:
-
 ```bash
 # Directly connect to localhost API without arguments
 vllm chat
@@ -60,8 +58,6 @@ vllm chat --quick "hi"
 
 Generate text completions based on the given prompt via the running API server.
 
-Examples:
-
 ```bash
 # Directly connect to localhost API without arguments
 vllm complete
@@ -73,6 +69,8 @@ vllm complete --url http://{vllm-serve-host}:{vllm-serve-port}/v1
 vllm complete --quick "The future of AI is"
 ```
 
+</details>
+
 ## bench
 
 Run benchmark tests for latency online serving throughput and offline inference throughput.
@@ -89,8 +87,6 @@ vllm bench {latency, serve, throughput}
 
 Benchmark the latency of a single batch of requests.
 
-Example:
-
 ```bash
 vllm bench latency \
     --model meta-llama/Llama-3.2-1B-Instruct \
@@ -104,8 +100,6 @@ vllm bench latency \
 
 Benchmark the online serving throughput.
 
-Example:
-
 ```bash
 vllm bench serve \
     --model meta-llama/Llama-3.2-1B-Instruct \
@@ -120,8 +114,6 @@ vllm bench serve \
 
 Benchmark offline inference throughput.
 
-Example:
-
 ```bash
 vllm bench throughput \
     --model meta-llama/Llama-3.2-1B-Instruct \
@@ -143,7 +135,8 @@ vllm collect-env
 
 Run batch prompts and write results to file.
 
-Examples:
+<details>
+<summary>Examples</summary>
 
 ```bash
 # Running with a local file
@@ -159,6 +152,8 @@ vllm run-batch \
     --model meta-llama/Meta-Llama-3-8B-Instruct
 ```
 
+</details>
+
 ## More Help
 
 For detailed options of any subcommand, use:

docs/configuration/conserving_memory.md

@@ -57,19 +57,21 @@ By default, we optimize model inference using CUDA graphs which take up extra me
 
 You can adjust `compilation_config` to achieve a better balance between inference speed and memory usage:
 
-```python
+??? Code
-from vllm import LLM
-from vllm.config import CompilationConfig, CompilationLevel
 
-llm = LLM(
+    ```python
+    from vllm import LLM
+    from vllm.config import CompilationConfig, CompilationLevel
+
+    llm = LLM(
         model="meta-llama/Llama-3.1-8B-Instruct",
         compilation_config=CompilationConfig(
             level=CompilationLevel.PIECEWISE,
             # By default, it goes up to max_num_seqs
             cudagraph_capture_sizes=[1, 2, 4, 8, 16],
         ),
-)
+    )
-```
+    ```
 
 You can disable graph capturing completely via the `enforce_eager` flag:
 
@@ -127,18 +129,20 @@ reduce the size of the processed multi-modal inputs, which in turn saves memory.
 
 Here are some examples:
 
-```python
+??? Code
-from vllm import LLM
 
-# Available for Qwen2-VL series models
+    ```python
-llm = LLM(model="Qwen/Qwen2.5-VL-3B-Instruct",
+    from vllm import LLM
+
+    # Available for Qwen2-VL series models
+    llm = LLM(model="Qwen/Qwen2.5-VL-3B-Instruct",
             mm_processor_kwargs={
                 "max_pixels": 768 * 768,  # Default is 1280 * 28 * 28
             })
 
-# Available for InternVL series models
+    # Available for InternVL series models
-llm = LLM(model="OpenGVLab/InternVL2-2B",
+    llm = LLM(model="OpenGVLab/InternVL2-2B",
             mm_processor_kwargs={
                 "max_dynamic_patch": 4,  # Default is 12
             })
-```
+    ```

docs/configuration/env_vars.md

@@ -7,6 +7,8 @@ vLLM uses the following environment variables to configure the system:
 
     All environment variables used by vLLM are prefixed with `VLLM_`. **Special care should be taken for Kubernetes users**: please do not name the service as `vllm`, otherwise environment variables set by Kubernetes might conflict with vLLM's environment variables, because [Kubernetes sets environment variables for each service with the capitalized service name as the prefix](https://kubernetes.io/docs/concepts/services-networking/service/#environment-variables).
 
-```python
+??? Code
---8<-- "vllm/envs.py:env-vars-definition"
+
-```
+    ```python
+    --8<-- "vllm/envs.py:env-vars-definition"
+    ```

docs/contributing/README.md

@@ -93,25 +93,27 @@ For additional features and advanced configurations, refer to the official [MkDo
 
 ## Testing
 
-```bash
+??? note "Commands"
-pip install -r requirements/dev.txt
 
-# Linting, formatting and static type checking
+    ```bash
-pre-commit install --hook-type pre-commit --hook-type commit-msg
+    pip install -r requirements/dev.txt
 
-# You can manually run pre-commit with
+    # Linting, formatting and static type checking
-pre-commit run --all-files
+    pre-commit install --hook-type pre-commit --hook-type commit-msg
 
-# To manually run something from CI that does not run
+    # You can manually run pre-commit with
-# locally by default, you can run:
+    pre-commit run --all-files
-pre-commit run mypy-3.9 --hook-stage manual --all-files
 
-# Unit tests
+    # To manually run something from CI that does not run
-pytest tests/
+    # locally by default, you can run:
+    pre-commit run mypy-3.9 --hook-stage manual --all-files
 
-# Run tests for a single test file with detailed output
+    # Unit tests
-pytest -s -v tests/test_logger.py
+    pytest tests/
-```
+
+    # Run tests for a single test file with detailed output
+    pytest -s -v tests/test_logger.py
+    ```
 
 !!! tip
     Since the <gh-file:docker/Dockerfile> ships with Python 3.12, all tests in CI (except `mypy`) are run with Python 3.12.

docs/contributing/model/basic.md

@@ -27,33 +27,35 @@ All vLLM modules within the model must include a `prefix` argument in their cons
 
 The initialization code should look like this:
 
-```python
+??? Code
-from torch import nn
-from vllm.config import VllmConfig
-from vllm.attention import Attention
 
-class MyAttention(nn.Module):
+    ```python
+    from torch import nn
+    from vllm.config import VllmConfig
+    from vllm.attention import Attention
+
+    class MyAttention(nn.Module):
         def __init__(self, vllm_config: VllmConfig, prefix: str):
             super().__init__()
             self.attn = Attention(prefix=f"{prefix}.attn")
 
-class MyDecoderLayer(nn.Module):
+    class MyDecoderLayer(nn.Module):
         def __init__(self, vllm_config: VllmConfig, prefix: str):
             super().__init__()
             self.self_attn = MyAttention(prefix=f"{prefix}.self_attn")
 
-class MyModel(nn.Module):
+    class MyModel(nn.Module):
         def __init__(self, vllm_config: VllmConfig, prefix: str):
             super().__init__()
             self.layers = nn.ModuleList(
                 [MyDecoderLayer(vllm_config, prefix=f"{prefix}.layers.{i}") for i in range(vllm_config.model_config.hf_config.num_hidden_layers)]
             )
 
-class MyModelForCausalLM(nn.Module):
+    class MyModelForCausalLM(nn.Module):
         def __init__(self, vllm_config: VllmConfig, prefix: str = ""):
             super().__init__()
             self.model = MyModel(vllm_config, prefix=f"{prefix}.model")
-```
+    ```
 
 ### Computation Code
 

docs/contributing/model/multimodal.md

@@ -25,6 +25,8 @@ Further update the model as follows:
 
 - Implement [get_multimodal_embeddings][vllm.model_executor.models.interfaces.SupportsMultiModal.get_multimodal_embeddings] that returns the embeddings from running the multimodal inputs through the multimodal tokenizer of the model. Below we provide a boilerplate of a typical implementation pattern, but feel free to adjust it to your own needs.
 
+    ??? Code
+
         ```python
         class YourModelForImage2Seq(nn.Module):
             ...
@@ -53,6 +55,8 @@ Further update the model as follows:
 
 - Implement [get_input_embeddings][vllm.model_executor.models.interfaces.SupportsMultiModal.get_input_embeddings] to merge `multimodal_embeddings` with text embeddings from the `input_ids`. If input processing for the model is implemented correctly (see sections below), then you can leverage the utility function we provide to easily merge the embeddings.
 
+    ??? Code
+
         ```python
         from .utils import merge_multimodal_embeddings
 
@@ -135,6 +139,8 @@ Assuming that the memory usage increases with the number of tokens, the dummy in
 
     Looking at the code of HF's `LlavaForConditionalGeneration`:
 
+    ??? Code
+
         ```python
         # https://github.com/huggingface/transformers/blob/v4.47.1/src/transformers/models/llava/modeling_llava.py#L530-L544
         n_image_tokens = (input_ids == self.config.image_token_index).sum().item()
@@ -157,6 +163,8 @@ Assuming that the memory usage increases with the number of tokens, the dummy in
     The number of placeholder feature tokens per image is `image_features.shape[1]`.
     `image_features` is calculated inside the `get_image_features` method:
 
+    ??? Code
+
         ```python
         # https://github.com/huggingface/transformers/blob/v4.47.1/src/transformers/models/llava/modeling_llava.py#L290-L300
         image_outputs = self.vision_tower(pixel_values, output_hidden_states=True)
@@ -193,6 +201,8 @@ Assuming that the memory usage increases with the number of tokens, the dummy in
 
     To find the sequence length, we turn to the code of `CLIPVisionEmbeddings`:
 
+    ??? Code
+
         ```python
         # https://github.com/huggingface/transformers/blob/v4.47.1/src/transformers/models/clip/modeling_clip.py#L247-L257
         target_dtype = self.patch_embedding.weight.dtype
@@ -218,6 +228,8 @@ Assuming that the memory usage increases with the number of tokens, the dummy in
 
     Overall, the number of placeholder feature tokens for an image can be calculated as:
 
+    ??? Code
+
         ```python
         def get_num_image_tokens(
             self,
@@ -241,6 +253,8 @@ Assuming that the memory usage increases with the number of tokens, the dummy in
     Notice that the number of image tokens doesn't depend on the image width and height.
     We can simply use a dummy `image_size` to calculate the multimodal profiling data:
 
+    ??? Code
+
         ```python
         # NOTE: In actuality, this is usually implemented as part of the
         # model's subclass of `BaseProcessingInfo`, but we show it as is
@@ -284,6 +298,8 @@ Assuming that the memory usage increases with the number of tokens, the dummy in
 
     Looking at the code of HF's `FuyuForCausalLM`:
 
+    ??? Code
+
         ```python
         # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/modeling_fuyu.py#L311-L322
         if image_patches is not None and past_key_values is None:
@@ -312,6 +328,8 @@ Assuming that the memory usage increases with the number of tokens, the dummy in
     In `FuyuImageProcessor.preprocess`, the images are resized and padded to the target `FuyuImageProcessor.size`,
     returning the dimensions after resizing (but before padding) as metadata.
 
+    ??? Code
+
         ```python
         # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/processing_fuyu.py#L541-L544
         image_encoding = self.image_processor.preprocess(images, **output_kwargs["images_kwargs"])
@@ -348,6 +366,8 @@ Assuming that the memory usage increases with the number of tokens, the dummy in
 
     In `FuyuImageProcessor.preprocess_with_tokenizer_info`, the images are split into patches based on this metadata:
 
+    ??? Code
+
         ```python
         # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/processing_fuyu.py#L417-L425
         model_image_input = self.image_processor.preprocess_with_tokenizer_info(
@@ -384,6 +404,8 @@ Assuming that the memory usage increases with the number of tokens, the dummy in
 
     The number of patches is in turn defined by `FuyuImageProcessor.get_num_patches`:
 
+    ??? Code
+
         ```python
         # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/image_processing_fuyu.py#L552-L562
         patch_size = patch_size if patch_size is not None else self.patch_size
@@ -419,6 +441,8 @@ Assuming that the memory usage increases with the number of tokens, the dummy in
 
     For the multimodal image profiling data, the logic is very similar to LLaVA:
 
+    ??? Code
+
         ```python
         def get_dummy_mm_data(
             self,
@@ -455,6 +479,7 @@ return a schema of the tensors outputted by the HF processor that are related to
     The output of `CLIPImageProcessor` is a simple tensor with shape
     `(num_images, num_channels, image_height, image_width)`:
 
+
     ```python
     # https://github.com/huggingface/transformers/blob/v4.47.1/src/transformers/models/clip/image_processing_clip.py#L339-L345
     images = [
@@ -505,6 +530,8 @@ return a schema of the tensors outputted by the HF processor that are related to
     In order to support the use of [MultiModalFieldConfig.batched][] like in LLaVA,
     we remove the extra batch dimension by overriding [BaseMultiModalProcessor._call_hf_processor][]:
 
+    ??? Code
+
         ```python
         def _call_hf_processor(
             self,
@@ -573,6 +600,8 @@ Each [PromptUpdate][vllm.multimodal.processing.PromptUpdate] instance specifies
     It simply repeats each input `image_token` a number of times equal to the number of placeholder feature tokens (`num_image_tokens`).
     Based on this, we override [_get_prompt_updates][vllm.multimodal.processing.BaseMultiModalProcessor._get_prompt_updates] as follows:
 
+    ??? Code
+
         ```python
         def _get_prompt_updates(
             self,
@@ -616,6 +645,8 @@ Each [PromptUpdate][vllm.multimodal.processing.PromptUpdate] instance specifies
 
     We define a helper function to return `ncols` and `nrows` directly:
 
+    ??? Code
+
         ```python
         def get_image_feature_grid_size(
             self,
@@ -644,6 +675,8 @@ Each [PromptUpdate][vllm.multimodal.processing.PromptUpdate] instance specifies
 
     Based on this, we can initially define our replacement tokens as:
 
+    ??? Code
+
         ```python
         def get_replacement(item_idx: int):
             images = mm_items.get_items("image", ImageProcessorItems)
@@ -662,6 +695,8 @@ Each [PromptUpdate][vllm.multimodal.processing.PromptUpdate] instance specifies
     However, this is not entirely correct. After `FuyuImageProcessor.preprocess_with_tokenizer_info` is called,
     a BOS token (`<s>`) is also added to the promopt:
 
+    ??? Code
+
         ```python
         # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/processing_fuyu.py#L417-L435
         model_image_input = self.image_processor.preprocess_with_tokenizer_info(
@@ -687,6 +722,8 @@ Each [PromptUpdate][vllm.multimodal.processing.PromptUpdate] instance specifies
     To assign the vision embeddings to only the image tokens, instead of a string
     you can return an instance of [PromptUpdateDetails][vllm.multimodal.processing.PromptUpdateDetails]:
 
+    ??? Code
+
         ```python
         hf_config = self.info.get_hf_config()
         bos_token_id = hf_config.bos_token_id  # `<s>`
@@ -712,6 +749,8 @@ Each [PromptUpdate][vllm.multimodal.processing.PromptUpdate] instance specifies
     Finally, noticing that the HF processor removes the `|ENDOFTEXT|` token from the tokenized prompt,
     we can search for it to conduct the replacement at the start of the string:
 
+    ??? Code
+
         ```python
         def _get_prompt_updates(
             self,

docs/contributing/profiling.md

@@ -97,11 +97,11 @@ to manually kill the profiler and generate your `nsys-rep` report.
 
 You can view these profiles either as summaries in the CLI, using `nsys stats [profile-file]`, or in the GUI by installing Nsight [locally following the directions here](https://developer.nvidia.com/nsight-systems/get-started).
 
-CLI example:
+??? CLI example
 
-```bash
+    ```bash
-nsys stats report1.nsys-rep
+    nsys stats report1.nsys-rep
-...
+    ...
     ** CUDA GPU Kernel Summary (cuda_gpu_kern_sum):
 
     Time (%)  Total Time (ns)  Instances   Avg (ns)     Med (ns)    Min (ns)  Max (ns)   StdDev (ns)                                                  Name                                                
@@ -115,8 +115,8 @@ nsys stats report1.nsys-rep
         2.6      587,283,113     37,824     15,526.7      3,008.0     2,719  2,517,756    139,091.1  std::enable_if<T2>(int)0&&vllm::_typeConvert<T1>::exists, void>::type vllm::fused_add_rms_norm_kern…
         1.9      418,362,605     18,912     22,121.5      3,871.0     3,328  2,523,870    175,248.2  void vllm::rotary_embedding_kernel<c10::BFloat16, (bool)1>(const long *, T1 *, T1 *, const T1 *, in…
         0.7      167,083,069     18,880      8,849.7      2,240.0     1,471  2,499,996    101,436.1  void vllm::reshape_and_cache_flash_kernel<__nv_bfloat16, __nv_bfloat16, (vllm::Fp8KVCacheDataType)0…
-... 
+    ... 
-```
+    ```
 
 GUI example:
 

docs/deployment/docker.md

@@ -97,10 +97,12 @@ of PyTorch Nightly and should be considered **experimental**. Using the flag `--
     flags to speed up build process. However, ensure your `max_jobs` is substantially larger than `nvcc_threads` to get the most benefits.
     Keep an eye on memory usage with parallel jobs as it can be substantial (see example below).
 
-```console
+??? Command
-# Example of building on Nvidia GH200 server. (Memory usage: ~15GB, Build time: ~1475s / ~25 min, Image size: 6.93GB)
+
-python3 use_existing_torch.py
+    ```console
-DOCKER_BUILDKIT=1 docker build . \
+    # Example of building on Nvidia GH200 server. (Memory usage: ~15GB, Build time: ~1475s / ~25 min, Image size: 6.93GB)
+    python3 use_existing_torch.py
+    DOCKER_BUILDKIT=1 docker build . \
     --file docker/Dockerfile \
     --target vllm-openai \
     --platform "linux/arm64" \
@@ -109,7 +111,7 @@ DOCKER_BUILDKIT=1 docker build . \
     --build-arg nvcc_threads=2 \
     --build-arg torch_cuda_arch_list="9.0 10.0+PTX" \
     --build-arg vllm_fa_cmake_gpu_arches="90-real"
-```
+    ```
 
 !!! note
     If you are building the `linux/arm64` image on a non-ARM host (e.g., an x86_64 machine), you need to ensure your system is set up for cross-compilation using QEMU. This allows your host machine to emulate ARM64 execution.

docs/deployment/frameworks/autogen.md

@@ -30,14 +30,16 @@ python -m vllm.entrypoints.openai.api_server \
 
 - Call it with AutoGen:
 
-```python
+??? Code
-import asyncio
+
-from autogen_core.models import UserMessage
+    ```python
-from autogen_ext.models.openai import OpenAIChatCompletionClient
+    import asyncio
-from autogen_core.models import ModelFamily
+    from autogen_core.models import UserMessage
+    from autogen_ext.models.openai import OpenAIChatCompletionClient
+    from autogen_core.models import ModelFamily
 
 
-async def main() -> None:
+    async def main() -> None:
         # Create a model client
         model_client = OpenAIChatCompletionClient(
             model="mistralai/Mistral-7B-Instruct-v0.2",
@@ -73,8 +75,8 @@ async def main() -> None:
         await model_client.close()
 
 
-asyncio.run(main())
+    asyncio.run(main())
-```
+    ```
 
 For details, see the tutorial:
 

docs/deployment/frameworks/cerebrium.md

@@ -34,12 +34,14 @@ vllm = "latest"
 
 Next, let us add our code to handle inference for the LLM of your choice (`mistralai/Mistral-7B-Instruct-v0.1` for this example), add the following code to your `main.py`:
 
-```python
+??? Code
-from vllm import LLM, SamplingParams
 
-llm = LLM(model="mistralai/Mistral-7B-Instruct-v0.1")
+    ```python
+    from vllm import LLM, SamplingParams
 
-def run(prompts: list[str], temperature: float = 0.8, top_p: float = 0.95):
+    llm = LLM(model="mistralai/Mistral-7B-Instruct-v0.1")
+
+    def run(prompts: list[str], temperature: float = 0.8, top_p: float = 0.95):
 
         sampling_params = SamplingParams(temperature=temperature, top_p=top_p)
         outputs = llm.generate(prompts, sampling_params)
@@ -52,7 +54,7 @@ def run(prompts: list[str], temperature: float = 0.8, top_p: float = 0.95):
             results.append({"prompt": prompt, "generated_text": generated_text})
 
         return {"results": results}
-```
+    ```
 
 Then, run the following code to deploy it to the cloud:
 
@@ -62,8 +64,10 @@ cerebrium deploy
 
 If successful, you should be returned a CURL command that you can call inference against. Just remember to end the url with the function name you are calling (in our case`/run`)
 
-```python
+??? Command
-curl -X POST https://api.cortex.cerebrium.ai/v4/p-xxxxxx/vllm/run \
+
+    ```python
+    curl -X POST https://api.cortex.cerebrium.ai/v4/p-xxxxxx/vllm/run \
     -H 'Content-Type: application/json' \
     -H 'Authorization: <JWT TOKEN>' \
     --data '{
@@ -74,12 +78,14 @@ curl -X POST https://api.cortex.cerebrium.ai/v4/p-xxxxxx/vllm/run \
         "The future of AI is"
     ]
     }'
-```
+    ```
 
 You should get a response like:
 
-```python
+??? Response
-{
+
+    ```python
+    {
         "run_id": "52911756-3066-9ae8-bcc9-d9129d1bd262",
         "result": {
             "result": [
@@ -102,7 +108,7 @@ You should get a response like:
             ]
         },
         "run_time_ms": 152.53663063049316
-}
+    }
-```
+    ```
 
 You now have an autoscaling endpoint where you only pay for the compute you use!

docs/deployment/frameworks/dstack.md

@@ -26,30 +26,34 @@ dstack init
 
 Next, to provision a VM instance with LLM of your choice (`NousResearch/Llama-2-7b-chat-hf` for this example), create the following `serve.dstack.yml` file for the dstack `Service`:
 
-```yaml
+??? Config
-type: service
 
-python: "3.11"
+    ```yaml
-env:
+    type: service
+
+    python: "3.11"
+    env:
         - MODEL=NousResearch/Llama-2-7b-chat-hf
-port: 8000
+    port: 8000
-resources:
+    resources:
         gpu: 24GB
-commands:
+    commands:
         - pip install vllm
         - vllm serve $MODEL --port 8000
-model:
+    model:
         format: openai
         type: chat
         name: NousResearch/Llama-2-7b-chat-hf
-```
+    ```
 
 Then, run the following CLI for provisioning:
 
-```console
+??? Command
-$ dstack run . -f serve.dstack.yml
 
-⠸ Getting run plan...
+    ```console
+    $ dstack run . -f serve.dstack.yml
+
+    ⠸ Getting run plan...
     Configuration  serve.dstack.yml
     Project        deep-diver-main
     User           deep-diver
@@ -66,24 +70,26 @@ $ dstack run . -f serve.dstack.yml
         ...
     Shown 3 of 193 offers, $5.876 max
 
-Continue? [y/n]: y
+    Continue? [y/n]: y
-⠙ Submitting run...
+    ⠙ Submitting run...
-⠏ Launching spicy-treefrog-1 (pulling)
+    ⠏ Launching spicy-treefrog-1 (pulling)
-spicy-treefrog-1 provisioning completed (running)
+    spicy-treefrog-1 provisioning completed (running)
-Service is published at ...
+    Service is published at ...
-```
+    ```
 
 After the provisioning, you can interact with the model by using the OpenAI SDK:
 
-```python
+??? Code
-from openai import OpenAI
 
-client = OpenAI(
+    ```python
+    from openai import OpenAI
+
+    client = OpenAI(
         base_url="https://gateway.<gateway domain>",
         api_key="<YOUR-DSTACK-SERVER-ACCESS-TOKEN>"
-)
+    )
 
-completion = client.chat.completions.create(
+    completion = client.chat.completions.create(
         model="NousResearch/Llama-2-7b-chat-hf",
         messages=[
             {
@@ -91,10 +97,10 @@ completion = client.chat.completions.create(
                 "content": "Compose a poem that explains the concept of recursion in programming.",
             }
         ]
-)
+    )
 
-print(completion.choices[0].message.content)
+    print(completion.choices[0].message.content)
-```
+    ```
 
 !!! note
     dstack automatically handles authentication on the gateway using dstack's tokens. Meanwhile, if you don't want to configure a gateway, you can provision dstack `Task` instead of `Service`. The `Task` is for development purpose only. If you want to know more about hands-on materials how to serve vLLM using dstack, check out [this repository](https://github.com/dstackai/dstack-examples/tree/main/deployment/vllm)

docs/deployment/frameworks/haystack.md

@@ -27,29 +27,29 @@ vllm serve mistralai/Mistral-7B-Instruct-v0.1
 
 - Use the `OpenAIGenerator` and `OpenAIChatGenerator` components in Haystack to query the vLLM server.
 
-```python
+??? Code
-from haystack.components.generators.chat import OpenAIChatGenerator
-from haystack.dataclasses import ChatMessage
-from haystack.utils import Secret
 
-generator = OpenAIChatGenerator(
+    ```python
+    from haystack.components.generators.chat import OpenAIChatGenerator
+    from haystack.dataclasses import ChatMessage
+    from haystack.utils import Secret
+
+    generator = OpenAIChatGenerator(
         # for compatibility with the OpenAI API, a placeholder api_key is needed
         api_key=Secret.from_token("VLLM-PLACEHOLDER-API-KEY"),
         model="mistralai/Mistral-7B-Instruct-v0.1",
         api_base_url="http://{your-vLLM-host-ip}:{your-vLLM-host-port}/v1",
         generation_kwargs = {"max_tokens": 512}
-)
+    )
 
-response = generator.run(
+    response = generator.run(
       messages=[ChatMessage.from_user("Hi. Can you help me plan my next trip to Italy?")]
-)
+    )
 
-print("-"*30)
+    print("-"*30)
-print(response)
+    print(response)
-print("-"*30)
+    print("-"*30)
-```
+    ```
-
-Output e.g.:
 
 ```console
 ------------------------------

docs/deployment/frameworks/litellm.md

@@ -34,21 +34,23 @@ vllm serve qwen/Qwen1.5-0.5B-Chat
 
 - Call it with litellm:
 
-```python
+??? Code
-import litellm 
 
-messages = [{ "content": "Hello, how are you?","role": "user"}]
+    ```python
+    import litellm 
 
-# hosted_vllm is prefix key word and necessary
+    messages = [{ "content": "Hello, how are you?","role": "user"}]
-response = litellm.completion(
+
+    # hosted_vllm is prefix key word and necessary
+    response = litellm.completion(
                 model="hosted_vllm/qwen/Qwen1.5-0.5B-Chat", # pass the vllm model name
                 messages=messages,
                 api_base="http://{your-vllm-server-host}:{your-vllm-server-port}/v1",
                 temperature=0.2,
                 max_tokens=80)
 
-print(response)
+    print(response)
-```
+    ```
 
 ### Embeddings
 

docs/deployment/frameworks/lws.md

@@ -17,12 +17,14 @@ vLLM can be deployed with [LWS](https://github.com/kubernetes-sigs/lws) on Kuber
 
 Deploy the following yaml file `lws.yaml`
 
-```yaml
+??? Yaml
-apiVersion: leaderworkerset.x-k8s.io/v1
+
-kind: LeaderWorkerSet
+    ```yaml
-metadata:
+    apiVersion: leaderworkerset.x-k8s.io/v1
+    kind: LeaderWorkerSet
+    metadata:
       name: vllm
-spec:
+    spec:
       replicas: 2
       leaderWorkerTemplate:
         size: 2
@@ -94,12 +96,12 @@ spec:
               emptyDir:
                 medium: Memory
                 sizeLimit: 15Gi
----
+    ---
-apiVersion: v1
+    apiVersion: v1
-kind: Service
+    kind: Service
-metadata:
+    metadata:
       name: vllm-leader
-spec:
+    spec:
       ports:
         - name: http
           port: 8080
@@ -109,7 +111,7 @@ spec:
         leaderworkerset.sigs.k8s.io/name: vllm
         role: leader
       type: ClusterIP
-```
+    ```
 
 ```bash
 kubectl apply -f lws.yaml
@@ -175,8 +177,10 @@ curl http://localhost:8080/v1/completions \
 
 The output should be similar to the following
 
-```text
+??? Output
-{
+
+    ```text
+    {
       "id": "cmpl-1bb34faba88b43f9862cfbfb2200949d",
       "object": "text_completion",
       "created": 1715138766,
@@ -195,5 +199,5 @@ The output should be similar to the following
         "total_tokens": 12,
         "completion_tokens": 7
       }
-}
+    }
-```
+    ```

docs/deployment/frameworks/skypilot.md

@@ -24,19 +24,21 @@ sky check
 
 See the vLLM SkyPilot YAML for serving, [serving.yaml](https://github.com/skypilot-org/skypilot/blob/master/llm/vllm/serve.yaml).
 
-```yaml
+??? Yaml
-resources:
+
+    ```yaml
+    resources:
       accelerators: {L4, A10g, A10, L40, A40, A100, A100-80GB} # We can use cheaper accelerators for 8B model.
       use_spot: True
       disk_size: 512  # Ensure model checkpoints can fit.
       disk_tier: best
       ports: 8081  # Expose to internet traffic.
 
-envs:
+    envs:
       MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct
       HF_TOKEN: <your-huggingface-token>  # Change to your own huggingface token, or use --env to pass.
 
-setup: |
+    setup: |
       conda create -n vllm python=3.10 -y
       conda activate vllm
 
@@ -45,7 +47,7 @@ setup: |
       pip install gradio openai
       pip install flash-attn==2.5.7
 
-run: |
+    run: |
       conda activate vllm
       echo 'Starting vllm api server...'
       python -u -m vllm.entrypoints.openai.api_server \
@@ -65,7 +67,7 @@ run: |
         --port 8811 \
         --model-url http://localhost:8081/v1 \
         --stop-token-ids 128009,128001
-```
+    ```
 
 Start the serving the Llama-3 8B model on any of the candidate GPUs listed (L4, A10g, ...):
 
@@ -93,8 +95,10 @@ HF_TOKEN="your-huggingface-token" \
 
 SkyPilot can scale up the service to multiple service replicas with built-in autoscaling, load-balancing and fault-tolerance. You can do it by adding a services section to the YAML file.
 
-```yaml
+??? Yaml
-service:
+
+    ```yaml
+    service:
       replicas: 2
       # An actual request for readiness probe.
       readiness_probe:
@@ -105,13 +109,12 @@ service:
           - role: user
             content: Hello! What is your name?
       max_completion_tokens: 1
-```
+    ```
 
-<details>
+??? Yaml
-<summary>Click to see the full recipe YAML</summary>
 
-```yaml
+    ```yaml
-service:
+    service:
       replicas: 2
       # An actual request for readiness probe.
       readiness_probe:
@@ -123,18 +126,18 @@ service:
               content: Hello! What is your name?
           max_completion_tokens: 1
 
-resources:
+    resources:
       accelerators: {L4, A10g, A10, L40, A40, A100, A100-80GB} # We can use cheaper accelerators for 8B model.
       use_spot: True
       disk_size: 512  # Ensure model checkpoints can fit.
       disk_tier: best
       ports: 8081  # Expose to internet traffic.
 
-envs:
+    envs:
       MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct
       HF_TOKEN: <your-huggingface-token>  # Change to your own huggingface token, or use --env to pass.
 
-setup: |
+    setup: |
       conda create -n vllm python=3.10 -y
       conda activate vllm
 
@@ -143,7 +146,7 @@ setup: |
       pip install gradio openai
       pip install flash-attn==2.5.7
 
-run: |
+    run: |
       conda activate vllm
       echo 'Starting vllm api server...'
       python -u -m vllm.entrypoints.openai.api_server \
@@ -152,9 +155,7 @@ run: |
         --trust-remote-code \
         --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \
         2>&1 | tee api_server.log
-```
+    ```
-
-</details>
 
 Start the serving the Llama-3 8B model on multiple replicas:
 
@@ -170,8 +171,7 @@ Wait until the service is ready:
 watch -n10 sky serve status vllm
 ```
 
-<details>
+Example outputs:
-<summary>Example outputs:</summary>
 
 ```console
 Services
@@ -184,13 +184,13 @@ vllm          1   1        xx.yy.zz.121  18 mins ago  1x GCP([Spot]{'L4': 1})  R
 vllm          2   1        xx.yy.zz.245  18 mins ago  1x GCP([Spot]{'L4': 1})  READY   us-east4
 ```
 
-</details>
-
 After the service is READY, you can find a single endpoint for the service and access the service with the endpoint:
 
-```console
+??? Commands
-ENDPOINT=$(sky serve status --endpoint 8081 vllm)
+
-curl -L http://$ENDPOINT/v1/chat/completions \
+    ```bash
+    ENDPOINT=$(sky serve status --endpoint 8081 vllm)
+    curl -L http://$ENDPOINT/v1/chat/completions \
       -H "Content-Type: application/json" \
       -d '{
         "model": "meta-llama/Meta-Llama-3-8B-Instruct",
@@ -206,7 +206,7 @@ curl -L http://$ENDPOINT/v1/chat/completions \
         ],
         "stop_token_ids": [128009,  128001]
       }'
-```
+    ```
 
 To enable autoscaling, you could replace the `replicas` with the following configs in `service`:
 
@@ -220,11 +220,10 @@ service:
 
 This will scale the service up to when the QPS exceeds 2 for each replica.
 
-<details>
+??? Yaml
-<summary>Click to see the full recipe YAML</summary>
 
-```yaml
+    ```yaml
-service:
+    service:
       replica_policy:
         min_replicas: 2
         max_replicas: 4
@@ -239,18 +238,18 @@ service:
               content: Hello! What is your name?
           max_completion_tokens: 1
 
-resources:
+    resources:
       accelerators: {L4, A10g, A10, L40, A40, A100, A100-80GB} # We can use cheaper accelerators for 8B model.
       use_spot: True
       disk_size: 512  # Ensure model checkpoints can fit.
       disk_tier: best
       ports: 8081  # Expose to internet traffic.
 
-envs:
+    envs:
       MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct
       HF_TOKEN: <your-huggingface-token>  # Change to your own huggingface token, or use --env to pass.
 
-setup: |
+    setup: |
       conda create -n vllm python=3.10 -y
       conda activate vllm
 
@@ -259,7 +258,7 @@ setup: |
       pip install gradio openai
       pip install flash-attn==2.5.7
 
-run: |
+    run: |
       conda activate vllm
       echo 'Starting vllm api server...'
       python -u -m vllm.entrypoints.openai.api_server \
@@ -268,9 +267,7 @@ run: |
         --trust-remote-code \
         --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \
         2>&1 | tee api_server.log
-```
+    ```
-
-</details>
 
 To update the service with the new config:
 
@@ -288,25 +285,24 @@ sky serve down vllm
 
 It is also possible to access the Llama-3 service with a separate GUI frontend, so the user requests send to the GUI will be load-balanced across replicas.
 
-<details>
+??? Yaml
-<summary>Click to see the full GUI YAML</summary>
 
-```yaml
+    ```yaml
-envs:
+    envs:
       MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct
       ENDPOINT: x.x.x.x:3031 # Address of the API server running vllm.
 
-resources:
+    resources:
       cpus: 2
 
-setup: |
+    setup: |
       conda create -n vllm python=3.10 -y
       conda activate vllm
 
       # Install Gradio for web UI.
       pip install gradio openai
 
-run: |
+    run: |
       conda activate vllm
       export PATH=$PATH:/sbin
 
@@ -317,9 +313,7 @@ run: |
         --port 8811 \
         --model-url http://$ENDPOINT/v1 \
         --stop-token-ids 128009,128001 | tee ~/gradio.log
-```
+    ```
-
-</details>
 
 1. Start the chat web UI:
 

docs/deployment/integrations/production-stack.md

@@ -60,10 +60,10 @@ And then you can send out a query to the OpenAI-compatible API to check the avai
 curl -o- http://localhost:30080/models
 ```
 
-Expected output:
+??? Output
 
-```json
+    ```json
-{
+    {
       "object": "list",
       "data": [
         {
@@ -74,8 +74,8 @@ Expected output:
           "root": null
         }
       ]
-}
+    }
-```
+    ```
 
 To send an actual chatting request, you can issue a curl request to the OpenAI `/completion` endpoint:
 
@@ -89,10 +89,10 @@ curl -X POST http://localhost:30080/completions \
   }'
 ```
 
-Expected output:
+??? Output
 
-```json
+    ```json
-{
+    {
       "id": "completion-id",
       "object": "text_completion",
       "created": 1737428424,
@@ -104,8 +104,8 @@ Expected output:
           "finish_reason": "length"
         }
       ]
-}
+    }
-```
+    ```
 
 ### Uninstall
 
@@ -121,8 +121,10 @@ sudo helm uninstall vllm
 
 The core vLLM production stack configuration is managed with YAML. Here is the example configuration used in the installation above:
 
-```yaml
+??? Yaml
-servingEngineSpec:
+
+    ```yaml
+    servingEngineSpec:
       runtimeClassName: ""
       modelSpec:
       - name: "opt125m"
@@ -137,7 +139,7 @@ servingEngineSpec:
         requestGPU: 1
 
         pvcStorage: "10Gi"
-```
+    ```
 
 In this YAML configuration:
 * **`modelSpec`** includes:

docs/deployment/k8s.md

@@ -29,39 +29,43 @@ Alternatively, you can deploy vLLM to Kubernetes using any of the following:
 
 First, create a Kubernetes PVC and Secret for downloading and storing Hugging Face model:
 
-```bash
+??? Config
-cat <<EOF |kubectl apply -f -
+
-apiVersion: v1
+    ```bash
-kind: PersistentVolumeClaim
+    cat <<EOF |kubectl apply -f -
-metadata:
+    apiVersion: v1
+    kind: PersistentVolumeClaim
+    metadata:
       name: vllm-models
-spec:
+    spec:
       accessModes:
         - ReadWriteOnce
       volumeMode: Filesystem
       resources:
         requests:
           storage: 50Gi
----
+    ---
-apiVersion: v1
+    apiVersion: v1
-kind: Secret
+    kind: Secret
-metadata:
+    metadata:
       name: hf-token-secret
-type: Opaque
+    type: Opaque
-data:
+    data:
       token: $(HF_TOKEN)
-EOF
+    EOF
-```
+    ```
 
 Next, start the vLLM server as a Kubernetes Deployment and Service:
 
-```bash
+??? Config
-cat <<EOF |kubectl apply -f -
+
-apiVersion: apps/v1
+    ```bash
-kind: Deployment
+    cat <<EOF |kubectl apply -f -
-metadata:
+    apiVersion: apps/v1
+    kind: Deployment
+    metadata:
       name: vllm-server
-spec:
+    spec:
       replicas: 1
       selector:
         matchLabels:
@@ -93,12 +97,12 @@ spec:
           - name: llama-storage
             persistentVolumeClaim:
               claimName: vllm-models
----
+    ---
-apiVersion: v1
+    apiVersion: v1
-kind: Service
+    kind: Service
-metadata:
+    metadata:
       name: vllm-server
-spec:
+    spec:
       selector:
         app.kubernetes.io/name: vllm
       ports:
@@ -106,8 +110,8 @@ spec:
         port: 8000
         targetPort: 8000
       type: ClusterIP
-EOF
+    EOF
-```
+    ```
 
 We can verify that the vLLM server has started successfully via the logs (this might take a couple of minutes to download the model):
 
@@ -128,6 +132,9 @@ INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
 
       PVC is used to store the model cache and it is optional, you can use hostPath or other storage options
 
+      <details>
+      <summary>Yaml</summary>
+
       ```yaml
       apiVersion: v1
       kind: PersistentVolumeClaim
@@ -144,6 +151,8 @@ INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
         volumeMode: Filesystem
       ```
 
+      </details>
+
       Secret is optional and only required for accessing gated models, you can skip this step if you are not using gated models
 
       ```yaml
@@ -163,6 +172,9 @@ INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
 
       NVIDIA GPU:
 
+      <details>
+      <summary>Yaml</summary>
+
       ```yaml
       apiVersion: apps/v1
       kind: Deployment
@@ -233,10 +245,15 @@ INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
                 periodSeconds: 5
       ```
 
+      </details>
+
       AMD GPU:
 
       You can refer to the `deployment.yaml` below if using AMD ROCm GPU like MI300X.
 
+      <details>
+      <summary>Yaml</summary>
+
       ```yaml
       apiVersion: apps/v1
       kind: Deployment
@@ -305,12 +322,17 @@ INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
                 mountPath: /dev/shm
       ```
 
+      </details>
+
       You can get the full example with steps and sample yaml files from <https://github.com/ROCm/k8s-device-plugin/tree/master/example/vllm-serve>.
 
 2. Create a Kubernetes Service for vLLM
 
       Next, create a Kubernetes Service file to expose the `mistral-7b` deployment:
 
+      <details>
+      <summary>Yaml</summary>
+
       ```yaml
       apiVersion: v1
       kind: Service
@@ -330,6 +352,8 @@ INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
         type: ClusterIP
       ```
 
+      </details>
+
 3. Deploy and Test
 
       Apply the deployment and service configurations using `kubectl apply -f <filename>`:

docs/deployment/nginx.md

@@ -36,13 +36,15 @@ docker build . -f Dockerfile.nginx --tag nginx-lb
 
 Create a file named `nginx_conf/nginx.conf`. Note that you can add as many servers as you'd like. In the below example we'll start with two. To add more, add another `server vllmN:8000 max_fails=3 fail_timeout=10000s;` entry to `upstream backend`.
 
-```console
+??? Config
-upstream backend {
+
+    ```console
+    upstream backend {
         least_conn;
         server vllm0:8000 max_fails=3 fail_timeout=10000s;
         server vllm1:8000 max_fails=3 fail_timeout=10000s;
-}
+    }
-server {
+    server {
         listen 80;
         location / {
             proxy_pass http://backend;
@@ -51,8 +53,8 @@ server {
             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
             proxy_set_header X-Forwarded-Proto $scheme;
         }
-}
+    }
-```
+    ```
 
 [](){ #nginxloadbalancer-nginx-vllm-container }
 
@@ -93,10 +95,12 @@ Notes:
 - The below example assumes GPU backend used. If you are using CPU backend, remove `--gpus device=ID`, add `VLLM_CPU_KVCACHE_SPACE` and `VLLM_CPU_OMP_THREADS_BIND` environment variables to the docker run command.
 - Adjust the model name that you want to use in your vLLM servers if you don't want to use `Llama-2-7b-chat-hf`.
 
-```console
+??? Commands
-mkdir -p ~/.cache/huggingface/hub/
+
-hf_cache_dir=~/.cache/huggingface/
+    ```console
-docker run \
+    mkdir -p ~/.cache/huggingface/hub/
+    hf_cache_dir=~/.cache/huggingface/
+    docker run \
         -itd \
         --ipc host \
         --network vllm_nginx \
@@ -106,7 +110,7 @@ docker run \
         -p 8081:8000 \
         --name vllm0 vllm \
         --model meta-llama/Llama-2-7b-chat-hf
-docker run \
+    docker run \
         -itd \
         --ipc host \
         --network vllm_nginx \
@@ -116,7 +120,7 @@ docker run \
         -p 8082:8000 \
         --name vllm1 vllm \
         --model meta-llama/Llama-2-7b-chat-hf
-```
+    ```
 
 !!! note
     If you are behind proxy, you can pass the proxy settings to the docker run command via `-e http_proxy=$http_proxy -e https_proxy=$https_proxy`.

docs/design/arch_overview.md

@@ -22,31 +22,33 @@ server.
 
 Here is a sample of `LLM` class usage:
 
-```python
+??? Code
-from vllm import LLM, SamplingParams
 
-# Define a list of input prompts
+    ```python
-prompts = [
+    from vllm import LLM, SamplingParams
+
+    # Define a list of input prompts
+    prompts = [
         "Hello, my name is",
         "The capital of France is",
         "The largest ocean is",
-]
+    ]
 
-# Define sampling parameters
+    # Define sampling parameters
-sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
+    sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
 
-# Initialize the LLM engine with the OPT-125M model
+    # Initialize the LLM engine with the OPT-125M model
-llm = LLM(model="facebook/opt-125m")
+    llm = LLM(model="facebook/opt-125m")
 
-# Generate outputs for the input prompts
+    # Generate outputs for the input prompts
-outputs = llm.generate(prompts, sampling_params)
+    outputs = llm.generate(prompts, sampling_params)
 
-# Print the generated outputs
+    # Print the generated outputs
-for output in outputs:
+    for output in outputs:
         prompt = output.prompt
         generated_text = output.outputs[0].text
         print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
-```
+    ```
 
 More API details can be found in the [Offline Inference](#offline-inference-api) section of the API docs.
 
@@ -178,6 +180,8 @@ vision-language model.
 
     To avoid accidentally passing incorrect arguments, the constructor is now keyword-only. This ensures that the constructor will raise an error if old configurations are passed. vLLM developers have already made this change for all models within vLLM. For out-of-tree registered models, developers need to update their models, for example by adding shim code to adapt the old constructor signature to the new one:
 
+    ??? Code
+
         ```python
         class MyOldModel(nn.Module):
             def __init__(

docs/design/kernel/paged_attention.md

@@ -448,9 +448,11 @@ elements of the entire head for all context tokens. However, overall,
 all results for output have been calculated but are just stored in
 different thread register memory.
 
-```cpp
+??? Code
-float* out_smem = reinterpret_cast<float*>(shared_mem);
+
-for (int i = NUM_WARPS; i > 1; i /= 2) {
+    ```cpp
+    float* out_smem = reinterpret_cast<float*>(shared_mem);
+    for (int i = NUM_WARPS; i > 1; i /= 2) {
         // Upper warps write to shared memory.
         ...
         float* dst = &out_smem[(warp_idx - mid) * HEAD_SIZE];
@@ -467,8 +469,8 @@ for (int i = NUM_WARPS; i > 1; i /= 2) {
         }
 
         // Write out the accs.
-}
+    }
-```
+    ```
 
 ## Output
 

docs/design/plugin_system.md

@@ -13,11 +13,13 @@ Plugins are user-registered code that vLLM executes. Given vLLM's architecture (
 
 vLLM's plugin system uses the standard Python `entry_points` mechanism. This mechanism allows developers to register functions in their Python packages for use by other packages. An example of a plugin:
 
-```python
+??? Code
-# inside `setup.py` file
-from setuptools import setup
 
-setup(name='vllm_add_dummy_model',
+    ```python
+    # inside `setup.py` file
+    from setuptools import setup
+
+    setup(name='vllm_add_dummy_model',
         version='0.1',
         packages=['vllm_add_dummy_model'],
         entry_points={
@@ -25,8 +27,8 @@ setup(name='vllm_add_dummy_model',
             ["register_dummy_model = vllm_add_dummy_model:register"]
         })
 
-# inside `vllm_add_dummy_model.py` file
+    # inside `vllm_add_dummy_model.py` file
-def register():
+    def register():
         from vllm import ModelRegistry
 
         if "MyLlava" not in ModelRegistry.get_supported_archs():
@@ -34,7 +36,7 @@ def register():
                 "MyLlava",
                 "vllm_add_dummy_model.my_llava:MyLlava",
             )
-```
+    ```
 
 For more information on adding entry points to your package, please check the [official documentation](https://setuptools.pypa.io/en/latest/userguide/entry_point.html).
 

docs/features/lora.md

@@ -29,24 +29,26 @@ We can now submit the prompts and call `llm.generate` with the `lora_request` pa
 of `LoRARequest` is a human identifiable name, the second parameter is a globally unique ID for the adapter and
 the third parameter is the path to the LoRA adapter.
 
-```python
+??? Code
-sampling_params = SamplingParams(
+
+    ```python
+    sampling_params = SamplingParams(
         temperature=0,
         max_tokens=256,
         stop=["[/assistant]"]
-)
+    )
 
-prompts = [
+    prompts = [
         "[user] Write a SQL query to answer the question based on the table schema.\n\n context: CREATE TABLE table_name_74 (icao VARCHAR, airport VARCHAR)\n\n question: Name the ICAO for lilongwe international airport [/user] [assistant]",
         "[user] Write a SQL query to answer the question based on the table schema.\n\n context: CREATE TABLE table_name_11 (nationality VARCHAR, elector VARCHAR)\n\n question: When Anchero Pantaleone was the elector what is under nationality? [/user] [assistant]",
-]
+    ]
 
-outputs = llm.generate(
+    outputs = llm.generate(
         prompts,
         sampling_params,
         lora_request=LoRARequest("sql_adapter", 1, sql_lora_path)
-)
+    )
-```
+    ```
 
 Check out <gh-file:examples/offline_inference/multilora_inference.py> for an example of how to use LoRA adapters with the async engine and how to use more advanced configuration options.
 
@@ -68,9 +70,11 @@ The server entrypoint accepts all other LoRA configuration parameters (`max_lora
 etc.), which will apply to all forthcoming requests. Upon querying the `/models` endpoint, we should see our LoRA along
 with its base model (if `jq` is not installed, you can follow [this guide](https://jqlang.org/download/) to install it.):
 
-```bash
+??? Command
-curl localhost:8000/v1/models | jq .
+
-{
+    ```bash
+    curl localhost:8000/v1/models | jq .
+    {
         "object": "list",
         "data": [
             {
@@ -84,8 +88,8 @@ curl localhost:8000/v1/models | jq .
                 ...
             }
         ]
-}
+    }
-```
+    ```
 
 Requests can specify the LoRA adapter as if it were any other model via the `model` request parameter. The requests will be
 processed according to the server-wide LoRA configuration (i.e. in parallel with base model requests, and potentially other
@@ -168,7 +172,7 @@ Alternatively, follow these example steps to implement your own plugin:
 
 1. Implement the LoRAResolver interface.
 
-    Example of a simple S3 LoRAResolver implementation:
+    ??? Example of a simple S3 LoRAResolver implementation
 
         ```python
         import os
@@ -234,10 +238,12 @@ The new format of `--lora-modules` is mainly to support the display of parent mo
 - The `parent` field of LoRA model `sql-lora` now links to its base model `meta-llama/Llama-2-7b-hf`. This correctly reflects the hierarchical relationship between the base model and the LoRA adapter.
 - The `root` field points to the artifact location of the lora adapter.
 
-```bash
+??? Command output
-$ curl http://localhost:8000/v1/models
 
-{
+    ```bash
+    $ curl http://localhost:8000/v1/models
+
+    {
         "object": "list",
         "data": [
             {
@@ -267,5 +273,5 @@ $ curl http://localhost:8000/v1/models
             ]
             }
         ]
-}
+    }
-```
+    ```

docs/features/multimodal_inputs.md

@@ -20,31 +20,33 @@ To input multi-modal data, follow this schema in [vllm.inputs.PromptType][]:
 
 You can pass a single image to the `'image'` field of the multi-modal dictionary, as shown in the following examples:
 
-```python
+??? Code
-from vllm import LLM
 
-llm = LLM(model="llava-hf/llava-1.5-7b-hf")
+    ```python
+    from vllm import LLM
 
-# Refer to the HuggingFace repo for the correct format to use
+    llm = LLM(model="llava-hf/llava-1.5-7b-hf")
-prompt = "USER: <image>\nWhat is the content of this image?\nASSISTANT:"
 
-# Load the image using PIL.Image
+    # Refer to the HuggingFace repo for the correct format to use
-image = PIL.Image.open(...)
+    prompt = "USER: <image>\nWhat is the content of this image?\nASSISTANT:"
 
-# Single prompt inference
+    # Load the image using PIL.Image
-outputs = llm.generate({
+    image = PIL.Image.open(...)
+
+    # Single prompt inference
+    outputs = llm.generate({
         "prompt": prompt,
         "multi_modal_data": {"image": image},
-})
+    })
 
-for o in outputs:
+    for o in outputs:
         generated_text = o.outputs[0].text
         print(generated_text)
 
-# Batch inference
+    # Batch inference
-image_1 = PIL.Image.open(...)
+    image_1 = PIL.Image.open(...)
-image_2 = PIL.Image.open(...)
+    image_2 = PIL.Image.open(...)
-outputs = llm.generate(
+    outputs = llm.generate(
         [
             {
                 "prompt": "USER: <image>\nWhat is the content of this image?\nASSISTANT:",
@@ -55,76 +57,80 @@ outputs = llm.generate(
                 "multi_modal_data": {"image": image_2},
             }
         ]
-)
+    )
 
-for o in outputs:
+    for o in outputs:
         generated_text = o.outputs[0].text
         print(generated_text)
-```
+    ```
 
 Full example: <gh-file:examples/offline_inference/vision_language.py>
 
 To substitute multiple images inside the same text prompt, you can pass in a list of images instead:
 
-```python
+??? Code
-from vllm import LLM
 
-llm = LLM(
+    ```python
+    from vllm import LLM
+
+    llm = LLM(
         model="microsoft/Phi-3.5-vision-instruct",
         trust_remote_code=True,  # Required to load Phi-3.5-vision
         max_model_len=4096,  # Otherwise, it may not fit in smaller GPUs
         limit_mm_per_prompt={"image": 2},  # The maximum number to accept
-)
+    )
 
-# Refer to the HuggingFace repo for the correct format to use
+    # Refer to the HuggingFace repo for the correct format to use
-prompt = "<|user|>\n<|image_1|>\n<|image_2|>\nWhat is the content of each image?<|end|>\n<|assistant|>\n"
+    prompt = "<|user|>\n<|image_1|>\n<|image_2|>\nWhat is the content of each image?<|end|>\n<|assistant|>\n"
 
-# Load the images using PIL.Image
+    # Load the images using PIL.Image
-image1 = PIL.Image.open(...)
+    image1 = PIL.Image.open(...)
-image2 = PIL.Image.open(...)
+    image2 = PIL.Image.open(...)
 
-outputs = llm.generate({
+    outputs = llm.generate({
         "prompt": prompt,
         "multi_modal_data": {
             "image": [image1, image2]
         },
-})
+    })
 
-for o in outputs:
+    for o in outputs:
         generated_text = o.outputs[0].text
         print(generated_text)
-```
+    ```
 
 Full example: <gh-file:examples/offline_inference/vision_language_multi_image.py>
 
 Multi-image input can be extended to perform video captioning. We show this with [Qwen2-VL](https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct) as it supports videos:
 
-```python
+??? Code
-from vllm import LLM
 
-# Specify the maximum number of frames per video to be 4. This can be changed.
+    ```python
-llm = LLM("Qwen/Qwen2-VL-2B-Instruct", limit_mm_per_prompt={"image": 4})
+    from vllm import LLM
 
-# Create the request payload.
+    # Specify the maximum number of frames per video to be 4. This can be changed.
-video_frames = ... # load your video making sure it only has the number of frames specified earlier.
+    llm = LLM("Qwen/Qwen2-VL-2B-Instruct", limit_mm_per_prompt={"image": 4})
-message = {
+
+    # Create the request payload.
+    video_frames = ... # load your video making sure it only has the number of frames specified earlier.
+    message = {
         "role": "user",
         "content": [
             {"type": "text", "text": "Describe this set of frames. Consider the frames to be a part of the same video."},
         ],
-}
+    }
-for i in range(len(video_frames)):
+    for i in range(len(video_frames)):
         base64_image = encode_image(video_frames[i]) # base64 encoding.
         new_image = {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}}
         message["content"].append(new_image)
 
-# Perform inference and log output.
+    # Perform inference and log output.
-outputs = llm.chat([message])
+    outputs = llm.chat([message])
 
-for o in outputs:
+    for o in outputs:
         generated_text = o.outputs[0].text
         print(generated_text)
-```
+    ```
 
 ### Video Inputs
 
@@ -144,68 +150,72 @@ Full example: <gh-file:examples/offline_inference/audio_language.py>
 To input pre-computed embeddings belonging to a data type (i.e. image, video, or audio) directly to the language model,
 pass a tensor of shape `(num_items, feature_size, hidden_size of LM)` to the corresponding field of the multi-modal dictionary.
 
-```python
+??? Code
-from vllm import LLM
 
-# Inference with image embeddings as input
+    ```python
-llm = LLM(model="llava-hf/llava-1.5-7b-hf")
+    from vllm import LLM
 
-# Refer to the HuggingFace repo for the correct format to use
+    # Inference with image embeddings as input
-prompt = "USER: <image>\nWhat is the content of this image?\nASSISTANT:"
+    llm = LLM(model="llava-hf/llava-1.5-7b-hf")
 
-# Embeddings for single image
+    # Refer to the HuggingFace repo for the correct format to use
-# torch.Tensor of shape (1, image_feature_size, hidden_size of LM)
+    prompt = "USER: <image>\nWhat is the content of this image?\nASSISTANT:"
-image_embeds = torch.load(...)
 
-outputs = llm.generate({
+    # Embeddings for single image
+    # torch.Tensor of shape (1, image_feature_size, hidden_size of LM)
+    image_embeds = torch.load(...)
+
+    outputs = llm.generate({
         "prompt": prompt,
         "multi_modal_data": {"image": image_embeds},
-})
+    })
 
-for o in outputs:
+    for o in outputs:
         generated_text = o.outputs[0].text
         print(generated_text)
-```
+    ```
 
 For Qwen2-VL and MiniCPM-V, we accept additional parameters alongside the embeddings:
 
-```python
+??? Code
-# Construct the prompt based on your model
-prompt = ...
 
-# Embeddings for multiple images
+    ```python
-# torch.Tensor of shape (num_images, image_feature_size, hidden_size of LM)
+    # Construct the prompt based on your model
-image_embeds = torch.load(...)
+    prompt = ...
 
-# Qwen2-VL
+    # Embeddings for multiple images
-llm = LLM("Qwen/Qwen2-VL-2B-Instruct", limit_mm_per_prompt={"image": 4})
+    # torch.Tensor of shape (num_images, image_feature_size, hidden_size of LM)
-mm_data = {
+    image_embeds = torch.load(...)
+
+    # Qwen2-VL
+    llm = LLM("Qwen/Qwen2-VL-2B-Instruct", limit_mm_per_prompt={"image": 4})
+    mm_data = {
         "image": {
             "image_embeds": image_embeds,
             # image_grid_thw is needed to calculate positional encoding.
             "image_grid_thw": torch.load(...),  # torch.Tensor of shape (1, 3),
         }
-}
+    }
 
-# MiniCPM-V
+    # MiniCPM-V
-llm = LLM("openbmb/MiniCPM-V-2_6", trust_remote_code=True, limit_mm_per_prompt={"image": 4})
+    llm = LLM("openbmb/MiniCPM-V-2_6", trust_remote_code=True, limit_mm_per_prompt={"image": 4})
-mm_data = {
+    mm_data = {
         "image": {
             "image_embeds": image_embeds,
             # image_sizes is needed to calculate details of the sliced image.
             "image_sizes": [image.size for image in images],  # list of image sizes
         }
-}
+    }
 
-outputs = llm.generate({
+    outputs = llm.generate({
         "prompt": prompt,
         "multi_modal_data": mm_data,
-})
+    })
 
-for o in outputs:
+    for o in outputs:
         generated_text = o.outputs[0].text
         print(generated_text)
-```
+    ```
 
 ## Online Serving
 
@@ -235,21 +245,23 @@ vllm serve microsoft/Phi-3.5-vision-instruct --task generate \
 
 Then, you can use the OpenAI client as follows:
 
-```python
+??? Code
-from openai import OpenAI
 
-openai_api_key = "EMPTY"
+    ```python
-openai_api_base = "http://localhost:8000/v1"
+    from openai import OpenAI
 
-client = OpenAI(
+    openai_api_key = "EMPTY"
+    openai_api_base = "http://localhost:8000/v1"
+
+    client = OpenAI(
         api_key=openai_api_key,
         base_url=openai_api_base,
-)
+    )
 
-# Single-image input inference
+    # Single-image input inference
-image_url = "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
+    image_url = "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
 
-chat_response = client.chat.completions.create(
+    chat_response = client.chat.completions.create(
         model="microsoft/Phi-3.5-vision-instruct",
         messages=[{
             "role": "user",
@@ -260,14 +272,14 @@ chat_response = client.chat.completions.create(
                 {"type": "image_url", "image_url": {"url": image_url}},
             ],
         }],
-)
+    )
-print("Chat completion output:", chat_response.choices[0].message.content)
+    print("Chat completion output:", chat_response.choices[0].message.content)
 
-# Multi-image input inference
+    # Multi-image input inference
-image_url_duck = "https://upload.wikimedia.org/wikipedia/commons/d/da/2015_Kaczka_krzy%C5%BCowka_w_wodzie_%28samiec%29.jpg"
+    image_url_duck = "https://upload.wikimedia.org/wikipedia/commons/d/da/2015_Kaczka_krzy%C5%BCowka_w_wodzie_%28samiec%29.jpg"
-image_url_lion = "https://upload.wikimedia.org/wikipedia/commons/7/77/002_The_lion_king_Snyggve_in_the_Serengeti_National_Park_Photo_by_Giles_Laurent.jpg"
+    image_url_lion = "https://upload.wikimedia.org/wikipedia/commons/7/77/002_The_lion_king_Snyggve_in_the_Serengeti_National_Park_Photo_by_Giles_Laurent.jpg"
 
-chat_response = client.chat.completions.create(
+    chat_response = client.chat.completions.create(
         model="microsoft/Phi-3.5-vision-instruct",
         messages=[{
             "role": "user",
@@ -277,9 +289,9 @@ chat_response = client.chat.completions.create(
                 {"type": "image_url", "image_url": {"url": image_url_lion}},
             ],
         }],
-)
+    )
-print("Chat completion output:", chat_response.choices[0].message.content)
+    print("Chat completion output:", chat_response.choices[0].message.content)
-```
+    ```
 
 Full example: <gh-file:examples/online_serving/openai_chat_completion_client_for_multimodal.py>
 
@@ -311,21 +323,23 @@ vllm serve llava-hf/llava-onevision-qwen2-0.5b-ov-hf --task generate --max-model
 
 Then, you can use the OpenAI client as follows:
 
-```python
+??? Code
-from openai import OpenAI
 
-openai_api_key = "EMPTY"
+    ```python
-openai_api_base = "http://localhost:8000/v1"
+    from openai import OpenAI
 
-client = OpenAI(
+    openai_api_key = "EMPTY"
+    openai_api_base = "http://localhost:8000/v1"
+
+    client = OpenAI(
         api_key=openai_api_key,
         base_url=openai_api_base,
-)
+    )
 
-video_url = "http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerFun.mp4"
+    video_url = "http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerFun.mp4"
 
-## Use video url in the payload
+    ## Use video url in the payload
-chat_completion_from_url = client.chat.completions.create(
+    chat_completion_from_url = client.chat.completions.create(
         messages=[{
             "role":
             "user",
@@ -344,11 +358,11 @@ chat_completion_from_url = client.chat.completions.create(
         }],
         model=model,
         max_completion_tokens=64,
-)
+    )
 
-result = chat_completion_from_url.choices[0].message.content
+    result = chat_completion_from_url.choices[0].message.content
-print("Chat completion output from image url:", result)
+    print("Chat completion output from image url:", result)
-```
+    ```
 
 Full example: <gh-file:examples/online_serving/openai_chat_completion_client_for_multimodal.py>
 
@@ -373,13 +387,15 @@ vllm serve fixie-ai/ultravox-v0_5-llama-3_2-1b
 
 Then, you can use the OpenAI client as follows:
 
-```python
+??? Code
-import base64
-import requests
-from openai import OpenAI
-from vllm.assets.audio import AudioAsset
 
-def encode_base64_content_from_url(content_url: str) -> str:
+    ```python
+    import base64
+    import requests
+    from openai import OpenAI
+    from vllm.assets.audio import AudioAsset
+
+    def encode_base64_content_from_url(content_url: str) -> str:
         """Encode a content retrieved from a remote url to base64 format."""
 
         with requests.get(content_url) as response:
@@ -388,19 +404,19 @@ def encode_base64_content_from_url(content_url: str) -> str:
 
         return result
 
-openai_api_key = "EMPTY"
+    openai_api_key = "EMPTY"
-openai_api_base = "http://localhost:8000/v1"
+    openai_api_base = "http://localhost:8000/v1"
 
-client = OpenAI(
+    client = OpenAI(
         api_key=openai_api_key,
         base_url=openai_api_base,
-)
+    )
 
-# Any format supported by librosa is supported
+    # Any format supported by librosa is supported
-audio_url = AudioAsset("winning_call").url
+    audio_url = AudioAsset("winning_call").url
-audio_base64 = encode_base64_content_from_url(audio_url)
+    audio_base64 = encode_base64_content_from_url(audio_url)
 
-chat_completion_from_base64 = client.chat.completions.create(
+    chat_completion_from_base64 = client.chat.completions.create(
         messages=[{
             "role": "user",
             "content": [
@@ -419,16 +435,18 @@ chat_completion_from_base64 = client.chat.completions.create(
         }],
         model=model,
         max_completion_tokens=64,
-)
+    )
 
-result = chat_completion_from_base64.choices[0].message.content
+    result = chat_completion_from_base64.choices[0].message.content
-print("Chat completion output from input audio:", result)
+    print("Chat completion output from input audio:", result)
-```
+    ```
 
 Alternatively, you can pass `audio_url`, which is the audio counterpart of `image_url` for image input:
 
-```python
+??? Code
-chat_completion_from_url = client.chat.completions.create(
+
+    ```python
+    chat_completion_from_url = client.chat.completions.create(
         messages=[{
             "role": "user",
             "content": [
@@ -446,11 +464,11 @@ chat_completion_from_url = client.chat.completions.create(
         }],
         model=model,
         max_completion_tokens=64,
-)
+    )
 
-result = chat_completion_from_url.choices[0].message.content
+    result = chat_completion_from_url.choices[0].message.content
-print("Chat completion output from audio url:", result)
+    print("Chat completion output from audio url:", result)
-```
+    ```
 
 Full example: <gh-file:examples/online_serving/openai_chat_completion_client_for_multimodal.py>
 
@@ -470,47 +488,49 @@ pass a tensor of shape to the corresponding field of the multi-modal dictionary.
 For image embeddings, you can pass the base64-encoded tensor to the `image_embeds` field.
 The following example demonstrates how to pass image embeddings to the OpenAI server:
 
-```python
+??? Code
-image_embedding = torch.load(...)
-grid_thw = torch.load(...) # Required by Qwen/Qwen2-VL-2B-Instruct
 
-buffer = io.BytesIO()
+    ```python
-torch.save(image_embedding, buffer)
+    image_embedding = torch.load(...)
-buffer.seek(0)
+    grid_thw = torch.load(...) # Required by Qwen/Qwen2-VL-2B-Instruct
-binary_data = buffer.read()
-base64_image_embedding = base64.b64encode(binary_data).decode('utf-8')
 
-client = OpenAI(
+    buffer = io.BytesIO()
+    torch.save(image_embedding, buffer)
+    buffer.seek(0)
+    binary_data = buffer.read()
+    base64_image_embedding = base64.b64encode(binary_data).decode('utf-8')
+
+    client = OpenAI(
         # defaults to os.environ.get("OPENAI_API_KEY")
         api_key=openai_api_key,
         base_url=openai_api_base,
-)
+    )
 
-# Basic usage - this is equivalent to the LLaVA example for offline inference
+    # Basic usage - this is equivalent to the LLaVA example for offline inference
-model = "llava-hf/llava-1.5-7b-hf"
+    model = "llava-hf/llava-1.5-7b-hf"
-embeds =  {
+    embeds =  {
         "type": "image_embeds",
         "image_embeds": f"{base64_image_embedding}" 
-}
+    }
 
-# Pass additional parameters (available to Qwen2-VL and MiniCPM-V)
+    # Pass additional parameters (available to Qwen2-VL and MiniCPM-V)
-model = "Qwen/Qwen2-VL-2B-Instruct"
+    model = "Qwen/Qwen2-VL-2B-Instruct"
-embeds =  {
+    embeds =  {
         "type": "image_embeds",
         "image_embeds": {
             "image_embeds": f"{base64_image_embedding}" , # Required
             "image_grid_thw": f"{base64_image_grid_thw}"  # Required by Qwen/Qwen2-VL-2B-Instruct
         },
-}
+    }
-model = "openbmb/MiniCPM-V-2_6"
+    model = "openbmb/MiniCPM-V-2_6"
-embeds =  {
+    embeds =  {
         "type": "image_embeds",
         "image_embeds": {
             "image_embeds": f"{base64_image_embedding}" , # Required
             "image_sizes": f"{base64_image_sizes}"  # Required by openbmb/MiniCPM-V-2_6
         },
-}
+    }
-chat_completion = client.chat.completions.create(
+    chat_completion = client.chat.completions.create(
         messages=[
         {"role": "system", "content": "You are a helpful assistant."},
         {"role": "user", "content": [
@@ -521,10 +541,10 @@ chat_completion = client.chat.completions.create(
             embeds,
             ],
         },
-],
+    ],
         model=model,
-)
+    )
-```
+    ```
 
 !!! note
     Only one message can contain `{"type": "image_embeds"}`.

docs/features/quantization/auto_awq.md

@@ -15,29 +15,31 @@ pip install autoawq
 
 After installing AutoAWQ, you are ready to quantize a model. Please refer to the [AutoAWQ documentation](https://casper-hansen.github.io/AutoAWQ/examples/#basic-quantization) for further details. Here is an example of how to quantize `mistralai/Mistral-7B-Instruct-v0.2`:
 
-```python
+??? Code
-from awq import AutoAWQForCausalLM
-from transformers import AutoTokenizer
 
-model_path = 'mistralai/Mistral-7B-Instruct-v0.2'
+    ```python
-quant_path = 'mistral-instruct-v0.2-awq'
+    from awq import AutoAWQForCausalLM
-quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }
+    from transformers import AutoTokenizer
 
-# Load model
+    model_path = 'mistralai/Mistral-7B-Instruct-v0.2'
-model = AutoAWQForCausalLM.from_pretrained(
+    quant_path = 'mistral-instruct-v0.2-awq'
+    quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }
+
+    # Load model
+    model = AutoAWQForCausalLM.from_pretrained(
         model_path, **{"low_cpu_mem_usage": True, "use_cache": False}
-)
+    )
-tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
+    tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
 
-# Quantize
+    # Quantize
-model.quantize(tokenizer, quant_config=quant_config)
+    model.quantize(tokenizer, quant_config=quant_config)
 
-# Save quantized model
+    # Save quantized model
-model.save_quantized(quant_path)
+    model.save_quantized(quant_path)
-tokenizer.save_pretrained(quant_path)
+    tokenizer.save_pretrained(quant_path)
 
-print(f'Model is quantized and saved at "{quant_path}"')
+    print(f'Model is quantized and saved at "{quant_path}"')
-```
+    ```
 
 To run an AWQ model with vLLM, you can use [TheBloke/Llama-2-7b-Chat-AWQ](https://huggingface.co/TheBloke/Llama-2-7b-Chat-AWQ) with the following command:
 
@@ -49,27 +51,29 @@ python examples/offline_inference/llm_engine_example.py \
 
 AWQ models are also supported directly through the LLM entrypoint:
 
-```python
+??? Code
-from vllm import LLM, SamplingParams
 
-# Sample prompts.
+    ```python
-prompts = [
+    from vllm import LLM, SamplingParams
+
+    # Sample prompts.
+    prompts = [
         "Hello, my name is",
         "The president of the United States is",
         "The capital of France is",
         "The future of AI is",
-]
+    ]
-# Create a sampling params object.
+    # Create a sampling params object.
-sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
+    sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
 
-# Create an LLM.
+    # Create an LLM.
-llm = LLM(model="TheBloke/Llama-2-7b-Chat-AWQ", quantization="AWQ")
+    llm = LLM(model="TheBloke/Llama-2-7b-Chat-AWQ", quantization="AWQ")
-# Generate texts from the prompts. The output is a list of RequestOutput objects
+    # Generate texts from the prompts. The output is a list of RequestOutput objects
-# that contain the prompt, generated text, and other information.
+    # that contain the prompt, generated text, and other information.
-outputs = llm.generate(prompts, sampling_params)
+    outputs = llm.generate(prompts, sampling_params)
-# Print the outputs.
+    # Print the outputs.
-for output in outputs:
+    for output in outputs:
         prompt = output.prompt
         generated_text = output.outputs[0].text
         print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
-```
+    ```

docs/features/quantization/bitblas.md

@@ -43,17 +43,19 @@ llm = LLM(
 
 ## Read gptq format checkpoint
 
-```python
+??? Code
-from vllm import LLM
-import torch
 
-# "hxbgsyxh/llama-13b-4bit-g-1" is a pre-quantized checkpoint.
+    ```python
-model_id = "hxbgsyxh/llama-13b-4bit-g-1"
+    from vllm import LLM
-llm = LLM(
+    import torch
+
+    # "hxbgsyxh/llama-13b-4bit-g-1" is a pre-quantized checkpoint.
+    model_id = "hxbgsyxh/llama-13b-4bit-g-1"
+    llm = LLM(
         model=model_id,
         dtype=torch.float16,
         trust_remote_code=True,
         quantization="bitblas",
         max_model_len=1024
-)
+    )
-```
+    ```

docs/features/quantization/fp8.md

@@ -58,22 +58,24 @@ For FP8 quantization, we can recover accuracy with simple RTN quantization. We r
 
 Since simple RTN does not require data for weight quantization and the activations are quantized dynamically, we do not need any calibration data for this quantization flow.
 
-```python
+??? Code
-from llmcompressor.transformers import oneshot
-from llmcompressor.modifiers.quantization import QuantizationModifier
 
-# Configure the simple PTQ quantization
+    ```python
-recipe = QuantizationModifier(
+    from llmcompressor.transformers import oneshot
+    from llmcompressor.modifiers.quantization import QuantizationModifier
+
+    # Configure the simple PTQ quantization
+    recipe = QuantizationModifier(
       targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head"])
 
-# Apply the quantization algorithm.
+    # Apply the quantization algorithm.
-oneshot(model=model, recipe=recipe)
+    oneshot(model=model, recipe=recipe)
 
-# Save the model: Meta-Llama-3-8B-Instruct-FP8-Dynamic
+    # Save the model: Meta-Llama-3-8B-Instruct-FP8-Dynamic
-SAVE_DIR = MODEL_ID.split("/")[1] + "-FP8-Dynamic"
+    SAVE_DIR = MODEL_ID.split("/")[1] + "-FP8-Dynamic"
-model.save_pretrained(SAVE_DIR)
+    model.save_pretrained(SAVE_DIR)
-tokenizer.save_pretrained(SAVE_DIR)
+    tokenizer.save_pretrained(SAVE_DIR)
-```
+    ```
 
 ### 3. Evaluating Accuracy
 

docs/features/quantization/gguf.md

@@ -41,11 +41,13 @@ vllm serve ./tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf \
 
 You can also use the GGUF model directly through the LLM entrypoint:
 
-```python
+??? Code
-from vllm import LLM, SamplingParams
 
-# In this script, we demonstrate how to pass input to the chat method:
+      ```python
-conversation = [
+      from vllm import LLM, SamplingParams
+
+      # In this script, we demonstrate how to pass input to the chat method:
+      conversation = [
          {
             "role": "system",
             "content": "You are a helpful assistant"
@@ -62,21 +64,21 @@ conversation = [
             "role": "user",
             "content": "Write an essay about the importance of higher education.",
          },
-]
+      ]
 
-# Create a sampling params object.
+      # Create a sampling params object.
-sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
+      sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
 
-# Create an LLM.
+      # Create an LLM.
-llm = LLM(model="./tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf",
+      llm = LLM(model="./tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf",
                tokenizer="TinyLlama/TinyLlama-1.1B-Chat-v1.0")
-# Generate texts from the prompts. The output is a list of RequestOutput objects
+      # Generate texts from the prompts. The output is a list of RequestOutput objects
-# that contain the prompt, generated text, and other information.
+      # that contain the prompt, generated text, and other information.
-outputs = llm.chat(conversation, sampling_params)
+      outputs = llm.chat(conversation, sampling_params)
 
-# Print the outputs.
+      # Print the outputs.
-for output in outputs:
+      for output in outputs:
          prompt = output.prompt
          generated_text = output.outputs[0].text
          print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
-```
+      ```

docs/features/quantization/gptqmodel.md

@@ -31,28 +31,30 @@ After installing GPTQModel, you are ready to quantize a model. Please refer to t
 
 Here is an example of how to quantize `meta-llama/Llama-3.2-1B-Instruct`:
 
-```python
+??? Code
-from datasets import load_dataset
-from gptqmodel import GPTQModel, QuantizeConfig
 
-model_id = "meta-llama/Llama-3.2-1B-Instruct"
+    ```python
-quant_path = "Llama-3.2-1B-Instruct-gptqmodel-4bit"
+    from datasets import load_dataset
+    from gptqmodel import GPTQModel, QuantizeConfig
 
-calibration_dataset = load_dataset(
+    model_id = "meta-llama/Llama-3.2-1B-Instruct"
+    quant_path = "Llama-3.2-1B-Instruct-gptqmodel-4bit"
+
+    calibration_dataset = load_dataset(
         "allenai/c4",
         data_files="en/c4-train.00001-of-01024.json.gz",
         split="train"
     ).select(range(1024))["text"]
 
-quant_config = QuantizeConfig(bits=4, group_size=128)
+    quant_config = QuantizeConfig(bits=4, group_size=128)
 
-model = GPTQModel.load(model_id, quant_config)
+    model = GPTQModel.load(model_id, quant_config)
 
-# increase `batch_size` to match gpu/vram specs to speed up quantization
+    # increase `batch_size` to match gpu/vram specs to speed up quantization
-model.quantize(calibration_dataset, batch_size=2)
+    model.quantize(calibration_dataset, batch_size=2)
 
-model.save(quant_path)
+    model.save(quant_path)
-```
+    ```
 
 ## Running a quantized model with vLLM
 
@@ -67,32 +69,34 @@ python examples/offline_inference/llm_engine_example.py \
 
 GPTQModel quantized models are also supported directly through the LLM entrypoint:
 
-```python
+??? Code
-from vllm import LLM, SamplingParams
 
-# Sample prompts.
+    ```python
-prompts = [
+    from vllm import LLM, SamplingParams
+
+    # Sample prompts.
+    prompts = [
         "Hello, my name is",
         "The president of the United States is",
         "The capital of France is",
         "The future of AI is",
-]
+    ]
 
-# Create a sampling params object.
+    # Create a sampling params object.
-sampling_params = SamplingParams(temperature=0.6, top_p=0.9)
+    sampling_params = SamplingParams(temperature=0.6, top_p=0.9)
 
-# Create an LLM.
+    # Create an LLM.
-llm = LLM(model="ModelCloud/DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2")
+    llm = LLM(model="ModelCloud/DeepSeek-R1-Distill-Qwen-7B-gptqmodel-4bit-vortex-v2")
 
-# Generate texts from the prompts. The output is a list of RequestOutput objects
+    # Generate texts from the prompts. The output is a list of RequestOutput objects
-# that contain the prompt, generated text, and other information.
+    # that contain the prompt, generated text, and other information.
-outputs = llm.generate(prompts, sampling_params)
+    outputs = llm.generate(prompts, sampling_params)
 
-# Print the outputs.
+    # Print the outputs.
-print("-"*50)
+    print("-"*50)
-for output in outputs:
+    for output in outputs:
         prompt = output.prompt
         generated_text = output.outputs[0].text
         print(f"Prompt: {prompt!r}\nGenerated text: {generated_text!r}")
         print("-"*50)
-```
+    ```

docs/features/quantization/int4.md

@@ -53,51 +53,55 @@ When quantizing weights to INT4, you need sample data to estimate the weight upd
 It's best to use calibration data that closely matches your deployment data.
 For a general-purpose instruction-tuned model, you can use a dataset like `ultrachat`:
 
-```python
+??? Code
-from datasets import load_dataset
 
-NUM_CALIBRATION_SAMPLES = 512
+    ```python
-MAX_SEQUENCE_LENGTH = 2048
+    from datasets import load_dataset
 
-# Load and preprocess the dataset
+    NUM_CALIBRATION_SAMPLES = 512
-ds = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft")
+    MAX_SEQUENCE_LENGTH = 2048
-ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES))
 
-def preprocess(example):
+    # Load and preprocess the dataset
+    ds = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft")
+    ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES))
+
+    def preprocess(example):
         return {"text": tokenizer.apply_chat_template(example["messages"], tokenize=False)}
-ds = ds.map(preprocess)
+    ds = ds.map(preprocess)
 
-def tokenize(sample):
+    def tokenize(sample):
         return tokenizer(sample["text"], padding=False, max_length=MAX_SEQUENCE_LENGTH, truncation=True, add_special_tokens=False)
-ds = ds.map(tokenize, remove_columns=ds.column_names)
+    ds = ds.map(tokenize, remove_columns=ds.column_names)
-```
+    ```
 
 ### 3. Applying Quantization
 
 Now, apply the quantization algorithms:
 
-```python
+??? Code
-from llmcompressor.transformers import oneshot
-from llmcompressor.modifiers.quantization import GPTQModifier
-from llmcompressor.modifiers.smoothquant import SmoothQuantModifier
 
-# Configure the quantization algorithms
+    ```python
-recipe = GPTQModifier(targets="Linear", scheme="W4A16", ignore=["lm_head"])
+    from llmcompressor.transformers import oneshot
+    from llmcompressor.modifiers.quantization import GPTQModifier
+    from llmcompressor.modifiers.smoothquant import SmoothQuantModifier
 
-# Apply quantization
+    # Configure the quantization algorithms
-oneshot(
+    recipe = GPTQModifier(targets="Linear", scheme="W4A16", ignore=["lm_head"])
+
+    # Apply quantization
+    oneshot(
         model=model,
         dataset=ds,
         recipe=recipe,
         max_seq_length=MAX_SEQUENCE_LENGTH,
         num_calibration_samples=NUM_CALIBRATION_SAMPLES,
-)
+    )
 
-# Save the compressed model: Meta-Llama-3-8B-Instruct-W4A16-G128
+    # Save the compressed model: Meta-Llama-3-8B-Instruct-W4A16-G128
-SAVE_DIR = MODEL_ID.split("/")[1] + "-W4A16-G128"
+    SAVE_DIR = MODEL_ID.split("/")[1] + "-W4A16-G128"
-model.save_pretrained(SAVE_DIR, save_compressed=True)
+    model.save_pretrained(SAVE_DIR, save_compressed=True)
-tokenizer.save_pretrained(SAVE_DIR)
+    tokenizer.save_pretrained(SAVE_DIR)
-```
+    ```
 
 This process creates a W4A16 model with weights quantized to 4-bit integers.
 
@@ -137,14 +141,16 @@ $ lm_eval --model vllm \
 
 The following is an example of an expanded quantization recipe you can tune to your own use case:
 
-```python
+??? Code
-from compressed_tensors.quantization import (
+
+    ```python
+    from compressed_tensors.quantization import (
         QuantizationArgs,
         QuantizationScheme,
         QuantizationStrategy,
         QuantizationType,
-) 
+    ) 
-recipe = GPTQModifier(
+    recipe = GPTQModifier(
         targets="Linear",
         config_groups={
             "config_group": QuantizationScheme(
@@ -163,8 +169,8 @@ recipe = GPTQModifier(
         ignore=["lm_head"],
         update_size=NUM_CALIBRATION_SAMPLES,
         dampening_frac=0.01
-)
+    )
-```
+    ```
 
 ## Troubleshooting and Support
 

docs/features/quantization/int8.md

@@ -54,54 +54,60 @@ When quantizing activations to INT8, you need sample data to estimate the activa
 It's best to use calibration data that closely matches your deployment data.
 For a general-purpose instruction-tuned model, you can use a dataset like `ultrachat`:
 
-```python
+??? Code
-from datasets import load_dataset
 
-NUM_CALIBRATION_SAMPLES = 512
+    ```python
-MAX_SEQUENCE_LENGTH = 2048
+    from datasets import load_dataset
 
-# Load and preprocess the dataset
+    NUM_CALIBRATION_SAMPLES = 512
-ds = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft")
+    MAX_SEQUENCE_LENGTH = 2048
-ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES))
 
-def preprocess(example):
+    # Load and preprocess the dataset
+    ds = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft")
+    ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES))
+
+    def preprocess(example):
         return {"text": tokenizer.apply_chat_template(example["messages"], tokenize=False)}
-ds = ds.map(preprocess)
+    ds = ds.map(preprocess)
 
-def tokenize(sample):
+    def tokenize(sample):
         return tokenizer(sample["text"], padding=False, max_length=MAX_SEQUENCE_LENGTH, truncation=True, add_special_tokens=False)
-ds = ds.map(tokenize, remove_columns=ds.column_names)
+    ds = ds.map(tokenize, remove_columns=ds.column_names)
-```
+    ```
+
+</details>
 
 ### 3. Applying Quantization
 
 Now, apply the quantization algorithms:
 
-```python
+??? Code
-from llmcompressor.transformers import oneshot
-from llmcompressor.modifiers.quantization import GPTQModifier
-from llmcompressor.modifiers.smoothquant import SmoothQuantModifier
 
-# Configure the quantization algorithms
+    ```python
-recipe = [
+    from llmcompressor.transformers import oneshot
+    from llmcompressor.modifiers.quantization import GPTQModifier
+    from llmcompressor.modifiers.smoothquant import SmoothQuantModifier
+
+    # Configure the quantization algorithms
+    recipe = [
         SmoothQuantModifier(smoothing_strength=0.8),
         GPTQModifier(targets="Linear", scheme="W8A8", ignore=["lm_head"]),
-]
+    ]
 
-# Apply quantization
+    # Apply quantization
-oneshot(
+    oneshot(
         model=model,
         dataset=ds,
         recipe=recipe,
         max_seq_length=MAX_SEQUENCE_LENGTH,
         num_calibration_samples=NUM_CALIBRATION_SAMPLES,
-)
+    )
 
-# Save the compressed model: Meta-Llama-3-8B-Instruct-W8A8-Dynamic-Per-Token
+    # Save the compressed model: Meta-Llama-3-8B-Instruct-W8A8-Dynamic-Per-Token
-SAVE_DIR = MODEL_ID.split("/")[1] + "-W8A8-Dynamic-Per-Token"
+    SAVE_DIR = MODEL_ID.split("/")[1] + "-W8A8-Dynamic-Per-Token"
-model.save_pretrained(SAVE_DIR, save_compressed=True)
+    model.save_pretrained(SAVE_DIR, save_compressed=True)
-tokenizer.save_pretrained(SAVE_DIR)
+    tokenizer.save_pretrained(SAVE_DIR)
-```
+    ```
 
 This process creates a W8A8 model with weights and activations quantized to 8-bit integers.
 

docs/features/quantization/modelopt.md

@@ -14,24 +14,26 @@ You can quantize HuggingFace models using the example scripts provided in the Te
 
 Below is an example showing how to quantize a model using modelopt's PTQ API:
 
-```python
+??? Code
-import modelopt.torch.quantization as mtq
-from transformers import AutoModelForCausalLM
 
-# Load the model from HuggingFace
+    ```python
-model = AutoModelForCausalLM.from_pretrained("<path_or_model_id>")
+    import modelopt.torch.quantization as mtq
+    from transformers import AutoModelForCausalLM
 
-# Select the quantization config, for example, FP8
+    # Load the model from HuggingFace
-config = mtq.FP8_DEFAULT_CFG
+    model = AutoModelForCausalLM.from_pretrained("<path_or_model_id>")
 
-# Define a forward loop function for calibration
+    # Select the quantization config, for example, FP8
-def forward_loop(model):
+    config = mtq.FP8_DEFAULT_CFG
+
+    # Define a forward loop function for calibration
+    def forward_loop(model):
         for data in calib_set:
             model(data)
 
-# PTQ with in-place replacement of quantized modules
+    # PTQ with in-place replacement of quantized modules
-model = mtq.quantize(model, config, forward_loop)
+    model = mtq.quantize(model, config, forward_loop)
-```
+    ```
 
 After the model is quantized, you can export it to a quantized checkpoint using the export API:
 
@@ -48,10 +50,12 @@ with torch.inference_mode():
 
 The quantized checkpoint can then be deployed with vLLM. As an example, the following code shows how to deploy `nvidia/Llama-3.1-8B-Instruct-FP8`, which is the FP8 quantized checkpoint derived from `meta-llama/Llama-3.1-8B-Instruct`, using vLLM:
 
-```python
+??? Code
-from vllm import LLM, SamplingParams
 
-def main():
+    ```python
+    from vllm import LLM, SamplingParams
+
+    def main():
 
         model_id = "nvidia/Llama-3.1-8B-Instruct-FP8"
         # Ensure you specify quantization='modelopt' when loading the modelopt checkpoint
@@ -73,6 +77,6 @@ def main():
             generated_text = output.outputs[0].text
             print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
 
-if __name__ == "__main__":
+    if __name__ == "__main__":
         main()
-```
+    ```

docs/features/quantization/quantized_kvcache.md

@@ -35,20 +35,22 @@ Studies have shown that FP8 E4M3 quantization typically only minimally degrades
 
 Here is an example of how to enable FP8 quantization:
 
-```python
+??? Code
-# To calculate kv cache scales on the fly enable the calculate_kv_scales
-# parameter
 
-from vllm import LLM, SamplingParams
+    ```python
+    # To calculate kv cache scales on the fly enable the calculate_kv_scales
+    # parameter
 
-sampling_params = SamplingParams(temperature=0.7, top_p=0.8)
+    from vllm import LLM, SamplingParams
-llm = LLM(model="meta-llama/Llama-2-7b-chat-hf",
+
+    sampling_params = SamplingParams(temperature=0.7, top_p=0.8)
+    llm = LLM(model="meta-llama/Llama-2-7b-chat-hf",
             kv_cache_dtype="fp8",
             calculate_kv_scales=True)
-prompt = "London is the capital of"
+    prompt = "London is the capital of"
-out = llm.generate(prompt, sampling_params)[0].outputs[0].text
+    out = llm.generate(prompt, sampling_params)[0].outputs[0].text
-print(out)
+    print(out)
-```
+    ```
 
 The `kv_cache_dtype` argument specifies the data type for KV cache storage:
 - `"auto"`: Uses the model's default "unquantized" data type
@@ -71,29 +73,31 @@ pip install llmcompressor
 
 Here's a complete example using `meta-llama/Llama-3.1-8B-Instruct` (most models can use this same pattern):
 
-```python
+??? Code
-from datasets import load_dataset
-from transformers import AutoModelForCausalLM, AutoTokenizer
-from llmcompressor.transformers import oneshot
 
-# Select model and load it
+    ```python
-MODEL_ID = "meta-llama/Llama-3.1-8B-Instruct"
+    from datasets import load_dataset
-model = AutoModelForCausalLM.from_pretrained(MODEL_ID, device_map="auto", torch_dtype="auto")
+    from transformers import AutoModelForCausalLM, AutoTokenizer
-tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
+    from llmcompressor.transformers import oneshot
 
-# Select calibration dataset
+    # Select model and load it
-DATASET_ID = "HuggingFaceH4/ultrachat_200k"
+    MODEL_ID = "meta-llama/Llama-3.1-8B-Instruct"
-DATASET_SPLIT = "train_sft"
+    model = AutoModelForCausalLM.from_pretrained(MODEL_ID, device_map="auto", torch_dtype="auto")
+    tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
 
-# Configure calibration parameters
+    # Select calibration dataset
-NUM_CALIBRATION_SAMPLES = 512  # 512 samples is a good starting point
+    DATASET_ID = "HuggingFaceH4/ultrachat_200k"
-MAX_SEQUENCE_LENGTH = 2048
+    DATASET_SPLIT = "train_sft"
 
-# Load and preprocess dataset
+    # Configure calibration parameters
-ds = load_dataset(DATASET_ID, split=DATASET_SPLIT)
+    NUM_CALIBRATION_SAMPLES = 512  # 512 samples is a good starting point
-ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES))
+    MAX_SEQUENCE_LENGTH = 2048
 
-def process_and_tokenize(example):
+    # Load and preprocess dataset
+    ds = load_dataset(DATASET_ID, split=DATASET_SPLIT)
+    ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES))
+
+    def process_and_tokenize(example):
         text = tokenizer.apply_chat_template(example["messages"], tokenize=False)
         return tokenizer(
             text,
@@ -103,11 +107,11 @@ def process_and_tokenize(example):
             add_special_tokens=False,
         )
 
-ds = ds.map(process_and_tokenize, remove_columns=ds.column_names)
+    ds = ds.map(process_and_tokenize, remove_columns=ds.column_names)
 
-# Configure quantization settings
+    # Configure quantization settings
-recipe = """
+    recipe = """
-quant_stage:
+    quant_stage:
         quant_modifiers:
             QuantizationModifier:
                 kv_cache_scheme:
@@ -116,22 +120,22 @@ quant_stage:
                     strategy: tensor
                     dynamic: false
                     symmetric: true
-"""
+    """
 
-# Apply quantization
+    # Apply quantization
-oneshot(
+    oneshot(
         model=model,
         dataset=ds,
         recipe=recipe,
         max_seq_length=MAX_SEQUENCE_LENGTH,
         num_calibration_samples=NUM_CALIBRATION_SAMPLES,
-)
+    )
 
-# Save quantized model: Llama-3.1-8B-Instruct-FP8-KV
+    # Save quantized model: Llama-3.1-8B-Instruct-FP8-KV
-SAVE_DIR = MODEL_ID.split("/")[1] + "-FP8-KV"
+    SAVE_DIR = MODEL_ID.split("/")[1] + "-FP8-KV"
-model.save_pretrained(SAVE_DIR, save_compressed=True)
+    model.save_pretrained(SAVE_DIR, save_compressed=True)
-tokenizer.save_pretrained(SAVE_DIR)
+    tokenizer.save_pretrained(SAVE_DIR)
-```
+    ```
 
 The above script will create a folder in your current directory containing your quantized model (e.g., `Llama-3.1-8B-Instruct-FP8-KV`) with calibrated scales.
 

docs/features/quantization/quark.md

@@ -42,20 +42,22 @@ The Quark quantization process can be listed for 5 steps as below:
 Quark uses [Transformers](https://huggingface.co/docs/transformers/en/index)
 to fetch model and tokenizer.
 
-```python
+??? Code
-from transformers import AutoTokenizer, AutoModelForCausalLM
 
-MODEL_ID = "meta-llama/Llama-2-70b-chat-hf"
+    ```python
-MAX_SEQ_LEN = 512
+    from transformers import AutoTokenizer, AutoModelForCausalLM
 
-model = AutoModelForCausalLM.from_pretrained(
+    MODEL_ID = "meta-llama/Llama-2-70b-chat-hf"
+    MAX_SEQ_LEN = 512
+
+    model = AutoModelForCausalLM.from_pretrained(
         MODEL_ID, device_map="auto", torch_dtype="auto",
-)
+    )
-model.eval()
+    model.eval()
 
-tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, model_max_length=MAX_SEQ_LEN)
+    tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, model_max_length=MAX_SEQ_LEN)
-tokenizer.pad_token = tokenizer.eos_token
+    tokenizer.pad_token = tokenizer.eos_token
-```
+    ```
 
 ### 2. Prepare the Calibration Dataloader
 
@@ -63,22 +65,24 @@ Quark uses the [PyTorch Dataloader](https://pytorch.org/tutorials/beginner/basic
 to load calibration data. For more details about how to use calibration datasets efficiently, please refer
 to [Adding Calibration Datasets](https://quark.docs.amd.com/latest/pytorch/calibration_datasets.html).
 
-```python
+??? Code
-from datasets import load_dataset
-from torch.utils.data import DataLoader
 
-BATCH_SIZE = 1
+    ```python
-NUM_CALIBRATION_DATA = 512
+    from datasets import load_dataset
+    from torch.utils.data import DataLoader
 
-# Load the dataset and get calibration data.
+    BATCH_SIZE = 1
-dataset = load_dataset("mit-han-lab/pile-val-backup", split="validation")
+    NUM_CALIBRATION_DATA = 512
-text_data = dataset["text"][:NUM_CALIBRATION_DATA]
 
-tokenized_outputs = tokenizer(text_data, return_tensors="pt",
+    # Load the dataset and get calibration data.
+    dataset = load_dataset("mit-han-lab/pile-val-backup", split="validation")
+    text_data = dataset["text"][:NUM_CALIBRATION_DATA]
+
+    tokenized_outputs = tokenizer(text_data, return_tensors="pt",
         padding=True, truncation=True, max_length=MAX_SEQ_LEN)
-calib_dataloader = DataLoader(tokenized_outputs['input_ids'],
+    calib_dataloader = DataLoader(tokenized_outputs['input_ids'],
         batch_size=BATCH_SIZE, drop_last=True)
-```
+    ```
 
 ### 3. Set the Quantization Configuration
 
@@ -94,42 +98,44 @@ kv-cache and the quantization algorithm is AutoSmoothQuant.
     AutoSmoothQuant config file for Llama is
     `examples/torch/language_modeling/llm_ptq/models/llama/autosmoothquant_config.json`.
 
-```python
+??? Code
-from quark.torch.quantization import (Config, QuantizationConfig,
+
+    ```python
+    from quark.torch.quantization import (Config, QuantizationConfig,
                                         FP8E4M3PerTensorSpec,
                                         load_quant_algo_config_from_file)
 
-# Define fp8/per-tensor/static spec.
+    # Define fp8/per-tensor/static spec.
-FP8_PER_TENSOR_SPEC = FP8E4M3PerTensorSpec(observer_method="min_max",
+    FP8_PER_TENSOR_SPEC = FP8E4M3PerTensorSpec(observer_method="min_max",
         is_dynamic=False).to_quantization_spec()
 
-# Define global quantization config, input tensors and weight apply FP8_PER_TENSOR_SPEC.
+    # Define global quantization config, input tensors and weight apply FP8_PER_TENSOR_SPEC.
-global_quant_config = QuantizationConfig(input_tensors=FP8_PER_TENSOR_SPEC,
+    global_quant_config = QuantizationConfig(input_tensors=FP8_PER_TENSOR_SPEC,
         weight=FP8_PER_TENSOR_SPEC)
 
-# Define quantization config for kv-cache layers, output tensors apply FP8_PER_TENSOR_SPEC.
+    # Define quantization config for kv-cache layers, output tensors apply FP8_PER_TENSOR_SPEC.
-KV_CACHE_SPEC = FP8_PER_TENSOR_SPEC
+    KV_CACHE_SPEC = FP8_PER_TENSOR_SPEC
-kv_cache_layer_names_for_llama = ["*k_proj", "*v_proj"]
+    kv_cache_layer_names_for_llama = ["*k_proj", "*v_proj"]
-kv_cache_quant_config = {name :
+    kv_cache_quant_config = {name :
         QuantizationConfig(input_tensors=global_quant_config.input_tensors,
                         weight=global_quant_config.weight,
                         output_tensors=KV_CACHE_SPEC)
         for name in kv_cache_layer_names_for_llama}
-layer_quant_config = kv_cache_quant_config.copy()
+    layer_quant_config = kv_cache_quant_config.copy()
 
-# Define algorithm config by config file.
+    # Define algorithm config by config file.
-LLAMA_AUTOSMOOTHQUANT_CONFIG_FILE =
+    LLAMA_AUTOSMOOTHQUANT_CONFIG_FILE =
         'examples/torch/language_modeling/llm_ptq/models/llama/autosmoothquant_config.json'
-algo_config = load_quant_algo_config_from_file(LLAMA_AUTOSMOOTHQUANT_CONFIG_FILE)
+    algo_config = load_quant_algo_config_from_file(LLAMA_AUTOSMOOTHQUANT_CONFIG_FILE)
 
-EXCLUDE_LAYERS = ["lm_head"]
+    EXCLUDE_LAYERS = ["lm_head"]
-quant_config = Config(
+    quant_config = Config(
         global_quant_config=global_quant_config,
         layer_quant_config=layer_quant_config,
         kv_cache_quant_config=kv_cache_quant_config,
         exclude=EXCLUDE_LAYERS,
         algo_config=algo_config)
-```
+    ```
 
 ### 4. Quantize the Model and Export
 
@@ -139,63 +145,67 @@ HuggingFace `safetensors`, you can refer to
 [HuggingFace format exporting](https://quark.docs.amd.com/latest/pytorch/export/quark_export_hf.html)
 for more exporting format details.
 
-```python
+??? Code
-import torch
-from quark.torch import ModelQuantizer, ModelExporter
-from quark.torch.export import ExporterConfig, JsonExporterConfig
 
-# Apply quantization.
+    ```python
-quantizer = ModelQuantizer(quant_config)
+    import torch
-quant_model = quantizer.quantize_model(model, calib_dataloader)
+    from quark.torch import ModelQuantizer, ModelExporter
+    from quark.torch.export import ExporterConfig, JsonExporterConfig
 
-# Freeze quantized model to export.
+    # Apply quantization.
-freezed_model = quantizer.freeze(model)
+    quantizer = ModelQuantizer(quant_config)
+    quant_model = quantizer.quantize_model(model, calib_dataloader)
 
-# Define export config.
+    # Freeze quantized model to export.
-LLAMA_KV_CACHE_GROUP = ["*k_proj", "*v_proj"]
+    freezed_model = quantizer.freeze(model)
-export_config = ExporterConfig(json_export_config=JsonExporterConfig())
-export_config.json_export_config.kv_cache_group = LLAMA_KV_CACHE_GROUP
 
-# Model: Llama-2-70b-chat-hf-w-fp8-a-fp8-kvcache-fp8-pertensor-autosmoothquant
+    # Define export config.
-EXPORT_DIR = MODEL_ID.split("/")[1] + "-w-fp8-a-fp8-kvcache-fp8-pertensor-autosmoothquant"
+    LLAMA_KV_CACHE_GROUP = ["*k_proj", "*v_proj"]
-exporter = ModelExporter(config=export_config, export_dir=EXPORT_DIR)
+    export_config = ExporterConfig(json_export_config=JsonExporterConfig())
-with torch.no_grad():
+    export_config.json_export_config.kv_cache_group = LLAMA_KV_CACHE_GROUP
+
+    # Model: Llama-2-70b-chat-hf-w-fp8-a-fp8-kvcache-fp8-pertensor-autosmoothquant
+    EXPORT_DIR = MODEL_ID.split("/")[1] + "-w-fp8-a-fp8-kvcache-fp8-pertensor-autosmoothquant"
+    exporter = ModelExporter(config=export_config, export_dir=EXPORT_DIR)
+    with torch.no_grad():
         exporter.export_safetensors_model(freezed_model,
             quant_config=quant_config, tokenizer=tokenizer)
-```
+    ```
 
 ### 5. Evaluation in vLLM
 
 Now, you can load and run the Quark quantized model directly through the LLM entrypoint:
 
-```python
+??? Code
-from vllm import LLM, SamplingParams
 
-# Sample prompts.
+    ```python
-prompts = [
+    from vllm import LLM, SamplingParams
+
+    # Sample prompts.
+    prompts = [
         "Hello, my name is",
         "The president of the United States is",
         "The capital of France is",
         "The future of AI is",
-]
+    ]
-# Create a sampling params object.
+    # Create a sampling params object.
-sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
+    sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
 
-# Create an LLM.
+    # Create an LLM.
-llm = LLM(model="Llama-2-70b-chat-hf-w-fp8-a-fp8-kvcache-fp8-pertensor-autosmoothquant",
+    llm = LLM(model="Llama-2-70b-chat-hf-w-fp8-a-fp8-kvcache-fp8-pertensor-autosmoothquant",
             kv_cache_dtype='fp8',quantization='quark')
-# Generate texts from the prompts. The output is a list of RequestOutput objects
+    # Generate texts from the prompts. The output is a list of RequestOutput objects
-# that contain the prompt, generated text, and other information.
+    # that contain the prompt, generated text, and other information.
-outputs = llm.generate(prompts, sampling_params)
+    outputs = llm.generate(prompts, sampling_params)
-# Print the outputs.
+    # Print the outputs.
-print("\nGenerated Outputs:\n" + "-" * 60)
+    print("\nGenerated Outputs:\n" + "-" * 60)
-for output in outputs:
+    for output in outputs:
         prompt = output.prompt
         generated_text = output.outputs[0].text
         print(f"Prompt:    {prompt!r}")
         print(f"Output:    {generated_text!r}")
         print("-" * 60)
-```
+    ```
 
 Or, you can use `lm_eval` to evaluate accuracy:
 

docs/features/quantization/torchao.md

@@ -15,26 +15,28 @@ pip install \
 ## Quantizing HuggingFace Models
 You can quantize your own huggingface model with torchao, e.g. [transformers](https://huggingface.co/docs/transformers/main/en/quantization/torchao) and [diffusers](https://huggingface.co/docs/diffusers/en/quantization/torchao), and save the checkpoint to huggingface hub like [this](https://huggingface.co/jerryzh168/llama3-8b-int8wo) with the following example code:
 
-```Python
+??? Code
-import torch
-from transformers import TorchAoConfig, AutoModelForCausalLM, AutoTokenizer
-from torchao.quantization import Int8WeightOnlyConfig
 
-model_name = "meta-llama/Meta-Llama-3-8B"
+    ```Python
-quantization_config = TorchAoConfig(Int8WeightOnlyConfig())
+    import torch
-quantized_model = AutoModelForCausalLM.from_pretrained(
+    from transformers import TorchAoConfig, AutoModelForCausalLM, AutoTokenizer
+    from torchao.quantization import Int8WeightOnlyConfig
+
+    model_name = "meta-llama/Meta-Llama-3-8B"
+    quantization_config = TorchAoConfig(Int8WeightOnlyConfig())
+    quantized_model = AutoModelForCausalLM.from_pretrained(
         model_name,
         torch_dtype="auto",
         device_map="auto",
         quantization_config=quantization_config
-)
+    )
-tokenizer = AutoTokenizer.from_pretrained(model_name)
+    tokenizer = AutoTokenizer.from_pretrained(model_name)
-input_text = "What are we having for dinner?"
+    input_text = "What are we having for dinner?"
-input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")
+    input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")
 
-hub_repo = # YOUR HUB REPO ID
+    hub_repo = # YOUR HUB REPO ID
-tokenizer.push_to_hub(hub_repo)
+    tokenizer.push_to_hub(hub_repo)
-quantized_model.push_to_hub(hub_repo, safe_serialization=False)
+    quantized_model.push_to_hub(hub_repo, safe_serialization=False)
-```
+    ```
 
 Alternatively, you can use the [TorchAO Quantization space](https://huggingface.co/spaces/medmekk/TorchAO_Quantization) for quantizing models with a simple UI.

docs/features/reasoning_outputs.md

@@ -33,34 +33,36 @@ vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \
 
 Next, make a request to the model that should return the reasoning content in the response.
 
-```python
+??? Code
-from openai import OpenAI
 
-# Modify OpenAI's API key and API base to use vLLM's API server.
+    ```python
-openai_api_key = "EMPTY"
+    from openai import OpenAI
-openai_api_base = "http://localhost:8000/v1"
 
-client = OpenAI(
+    # Modify OpenAI's API key and API base to use vLLM's API server.
+    openai_api_key = "EMPTY"
+    openai_api_base = "http://localhost:8000/v1"
+
+    client = OpenAI(
         api_key=openai_api_key,
         base_url=openai_api_base,
-)
+    )
 
-models = client.models.list()
+    models = client.models.list()
-model = models.data[0].id
+    model = models.data[0].id
 
-# Round 1
+    # Round 1
-messages = [{"role": "user", "content": "9.11 and 9.8, which is greater?"}]
+    messages = [{"role": "user", "content": "9.11 and 9.8, which is greater?"}]
-# For granite, add: `extra_body={"chat_template_kwargs": {"thinking": True}}`
+    # For granite, add: `extra_body={"chat_template_kwargs": {"thinking": True}}`
-# For Qwen3 series, if you want to disable thinking in reasoning mode, add:
+    # For Qwen3 series, if you want to disable thinking in reasoning mode, add:
-# extra_body={"chat_template_kwargs": {"enable_thinking": False}}
+    # extra_body={"chat_template_kwargs": {"enable_thinking": False}}
-response = client.chat.completions.create(model=model, messages=messages)
+    response = client.chat.completions.create(model=model, messages=messages)
 
-reasoning_content = response.choices[0].message.reasoning_content
+    reasoning_content = response.choices[0].message.reasoning_content
-content = response.choices[0].message.content
+    content = response.choices[0].message.content
 
-print("reasoning_content:", reasoning_content)
+    print("reasoning_content:", reasoning_content)
-print("content:", content)
+    print("content:", content)
-```
+    ```
 
 The `reasoning_content` field contains the reasoning steps that led to the final conclusion, while the `content` field contains the final conclusion.
 
@@ -68,8 +70,10 @@ The `reasoning_content` field contains the reasoning steps that led to the final
 
 Streaming chat completions are also supported for reasoning models. The `reasoning_content` field is available in the `delta` field in [chat completion response chunks](https://platform.openai.com/docs/api-reference/chat/streaming).
 
-```json
+??? Json
-{
+
+    ```json
+    {
         "id": "chatcmpl-123",
         "object": "chat.completion.chunk",
         "created": 1694268190,
@@ -86,39 +90,41 @@ Streaming chat completions are also supported for reasoning models. The `reasoni
                 "finish_reason": null
             }
         ]
-}
+    }
-```
+    ```
 
 OpenAI Python client library does not officially support `reasoning_content` attribute for streaming output. But the client supports extra attributes in the response. You can use `hasattr` to check if the `reasoning_content` attribute is present in the response. For example:
 
-```python
+??? Code
-from openai import OpenAI
 
-# Modify OpenAI's API key and API base to use vLLM's API server.
+    ```python
-openai_api_key = "EMPTY"
+    from openai import OpenAI
-openai_api_base = "http://localhost:8000/v1"
 
-client = OpenAI(
+    # Modify OpenAI's API key and API base to use vLLM's API server.
+    openai_api_key = "EMPTY"
+    openai_api_base = "http://localhost:8000/v1"
+
+    client = OpenAI(
         api_key=openai_api_key,
         base_url=openai_api_base,
-)
+    )
 
-models = client.models.list()
+    models = client.models.list()
-model = models.data[0].id
+    model = models.data[0].id
 
-messages = [{"role": "user", "content": "9.11 and 9.8, which is greater?"}]
+    messages = [{"role": "user", "content": "9.11 and 9.8, which is greater?"}]
-# For granite, add: `extra_body={"chat_template_kwargs": {"thinking": True}}`
+    # For granite, add: `extra_body={"chat_template_kwargs": {"thinking": True}}`
-# For Qwen3 series, if you want to disable thinking in reasoning mode, add:
+    # For Qwen3 series, if you want to disable thinking in reasoning mode, add:
-# extra_body={"chat_template_kwargs": {"enable_thinking": False}}
+    # extra_body={"chat_template_kwargs": {"enable_thinking": False}}
-stream = client.chat.completions.create(model=model,
+    stream = client.chat.completions.create(model=model,
                                             messages=messages,
                                             stream=True)
 
-print("client: Start streaming chat completions...")
+    print("client: Start streaming chat completions...")
-printed_reasoning_content = False
+    printed_reasoning_content = False
-printed_content = False
+    printed_content = False
 
-for chunk in stream:
+    for chunk in stream:
         reasoning_content = None
         content = None
         # Check the content is reasoning_content or content
@@ -138,7 +144,7 @@ for chunk in stream:
                 print("\ncontent:", end="", flush=True)
             # Extract and print the content
             print(content, end="", flush=True)
-```
+    ```
 
 Remember to check whether the `reasoning_content` exists in the response before accessing it. You could checkout the [example](https://github.com/vllm-project/vllm/blob/main/examples/online_serving/openai_chat_completion_with_reasoning_streaming.py).
 
@@ -146,12 +152,14 @@ Remember to check whether the `reasoning_content` exists in the response before
 
 The reasoning content is also available when both tool calling and the reasoning parser are enabled. Additionally, tool calling only parses functions from the `content` field, not from the `reasoning_content`.
 
-```python
+??? Code
-from openai import OpenAI
 
-client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")
+    ```python
+    from openai import OpenAI
 
-tools = [{
+    client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")
+
+    tools = [{
         "type": "function",
         "function": {
             "name": "get_weather",
@@ -165,22 +173,22 @@ tools = [{
                 "required": ["location", "unit"]
             }
         }
-}]
+    }]
 
-response = client.chat.completions.create(
+    response = client.chat.completions.create(
         model=client.models.list().data[0].id,
         messages=[{"role": "user", "content": "What's the weather like in San Francisco?"}],
         tools=tools,
         tool_choice="auto"
-)
+    )
 
-print(response)
+    print(response)
-tool_call = response.choices[0].message.tool_calls[0].function
+    tool_call = response.choices[0].message.tool_calls[0].function
 
-print(f"reasoning_content: {response.choices[0].message.reasoning_content}")
+    print(f"reasoning_content: {response.choices[0].message.reasoning_content}")
-print(f"Function called: {tool_call.name}")
+    print(f"Function called: {tool_call.name}")
-print(f"Arguments: {tool_call.arguments}")
+    print(f"Arguments: {tool_call.arguments}")
-```
+    ```
 
 For more examples, please refer to <gh-file:examples/online_serving/openai_chat_completion_tool_calls_with_reasoning.py>.
 
@@ -192,18 +200,20 @@ For more examples, please refer to <gh-file:examples/online_serving/openai_chat_
 
 You can add a new `ReasoningParser` similar to <gh-file:vllm/reasoning/deepseek_r1_reasoning_parser.py>.
 
-```python
+??? Code
-# import the required packages
 
-from vllm.reasoning import ReasoningParser, ReasoningParserManager
+    ```python
-from vllm.entrypoints.openai.protocol import (ChatCompletionRequest,
+    # import the required packages
+
+    from vllm.reasoning import ReasoningParser, ReasoningParserManager
+    from vllm.entrypoints.openai.protocol import (ChatCompletionRequest,
                                                 DeltaMessage)
 
-# define a reasoning parser and register it to vllm
+    # define a reasoning parser and register it to vllm
-# the name list in register_module can be used
+    # the name list in register_module can be used
-# in --reasoning-parser.
+    # in --reasoning-parser.
-@ReasoningParserManager.register_module(["example"])
+    @ReasoningParserManager.register_module(["example"])
-class ExampleParser(ReasoningParser):
+    class ExampleParser(ReasoningParser):
         def __init__(self, tokenizer: AnyTokenizer):
             super().__init__(tokenizer)
 
@@ -244,13 +254,15 @@ class ExampleParser(ReasoningParser):
             tuple[Optional[str], Optional[str]]
                 A tuple containing the reasoning content and the content.
             """
-```
+    ```
 
 Additionally, to enable structured output, you'll need to create a new `Reasoner` similar to the one in <gh-file:vllm/reasoning/deepseek_r1_reasoning_parser.py>.
 
-```python
+??? Code
-@dataclass
+
-class DeepSeekReasoner(Reasoner):
+    ```python
+    @dataclass
+    class DeepSeekReasoner(Reasoner):
         """
         Reasoner for DeepSeek R series models.
         """
@@ -270,7 +282,7 @@ class DeepSeekReasoner(Reasoner):
         def is_reasoning_end(self, input_ids: list[int]) -> bool:
             return self.end_token_id in input_ids
         ...
-```
+    ```
 
 The structured output engine like [xgrammar](https://github.com/mlc-ai/xgrammar) will use `end_token_id` to check if the reasoning content is present in the model output and skip the structured output if it is the case.
 

docs/features/spec_decode.md

@@ -18,29 +18,31 @@ Speculative decoding is a technique which improves inter-token latency in memory
 
 The following code configures vLLM in an offline mode to use speculative decoding with a draft model, speculating 5 tokens at a time.
 
-```python
+??? Code
-from vllm import LLM, SamplingParams
 
-prompts = [
+    ```python
+    from vllm import LLM, SamplingParams
+
+    prompts = [
         "The future of AI is",
-]
+    ]
-sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
+    sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
 
-llm = LLM(
+    llm = LLM(
         model="facebook/opt-6.7b",
         tensor_parallel_size=1,
         speculative_config={
             "model": "facebook/opt-125m",
             "num_speculative_tokens": 5,
         },
-)
+    )
-outputs = llm.generate(prompts, sampling_params)
+    outputs = llm.generate(prompts, sampling_params)
 
-for output in outputs:
+    for output in outputs:
         prompt = output.prompt
         generated_text = output.outputs[0].text
         print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
-```
+    ```
 
 To perform the same with an online mode launch the server:
 
@@ -60,54 +62,58 @@ python -m vllm.entrypoints.openai.api_server \
 
 Then use a client:
 
-```python
+??? Code
-from openai import OpenAI
 
-# Modify OpenAI's API key and API base to use vLLM's API server.
+    ```python
-openai_api_key = "EMPTY"
+    from openai import OpenAI
-openai_api_base = "http://localhost:8000/v1"
 
-client = OpenAI(
+    # Modify OpenAI's API key and API base to use vLLM's API server.
+    openai_api_key = "EMPTY"
+    openai_api_base = "http://localhost:8000/v1"
+
+    client = OpenAI(
         # defaults to os.environ.get("OPENAI_API_KEY")
         api_key=openai_api_key,
         base_url=openai_api_base,
-)
+    )
 
-models = client.models.list()
+    models = client.models.list()
-model = models.data[0].id
+    model = models.data[0].id
 
-# Completion API
+    # Completion API
-stream = False
+    stream = False
-completion = client.completions.create(
+    completion = client.completions.create(
         model=model,
         prompt="The future of AI is",
         echo=False,
         n=1,
         stream=stream,
-)
+    )
 
-print("Completion results:")
+    print("Completion results:")
-if stream:
+    if stream:
         for c in completion:
             print(c)
-else:
+    else:
         print(completion)
-```
+    ```
 
 ## Speculating by matching n-grams in the prompt
 
 The following code configures vLLM to use speculative decoding where proposals are generated by
 matching n-grams in the prompt. For more information read [this thread.](https://x.com/joao_gante/status/1747322413006643259)
 
-```python
+??? Code
-from vllm import LLM, SamplingParams
 
-prompts = [
+    ```python
+    from vllm import LLM, SamplingParams
+
+    prompts = [
         "The future of AI is",
-]
+    ]
-sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
+    sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
 
-llm = LLM(
+    llm = LLM(
         model="facebook/opt-6.7b",
         tensor_parallel_size=1,
         speculative_config={
@@ -115,14 +121,14 @@ llm = LLM(
             "num_speculative_tokens": 5,
             "prompt_lookup_max": 4,
         },
-)
+    )
-outputs = llm.generate(prompts, sampling_params)
+    outputs = llm.generate(prompts, sampling_params)
 
-for output in outputs:
+    for output in outputs:
         prompt = output.prompt
         generated_text = output.outputs[0].text
         print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
-```
+    ```
 
 ## Speculating using MLP speculators
 
@@ -131,29 +137,31 @@ draft models that conditioning draft predictions on both context vectors and sam
 For more information see [this blog](https://pytorch.org/blog/hitchhikers-guide-speculative-decoding/) or
 [this technical report](https://arxiv.org/abs/2404.19124).
 
-```python
+??? Code
-from vllm import LLM, SamplingParams
 
-prompts = [
+    ```python
+    from vllm import LLM, SamplingParams
+
+    prompts = [
         "The future of AI is",
-]
+    ]
-sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
+    sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
 
-llm = LLM(
+    llm = LLM(
         model="meta-llama/Meta-Llama-3.1-70B-Instruct",
         tensor_parallel_size=4,
         speculative_config={
             "model": "ibm-ai-platform/llama3-70b-accelerator",
             "draft_tensor_parallel_size": 1,
         },
-)
+    )
-outputs = llm.generate(prompts, sampling_params)
+    outputs = llm.generate(prompts, sampling_params)
 
-for output in outputs:
+    for output in outputs:
         prompt = output.prompt
         generated_text = output.outputs[0].text
         print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
-```
+    ```
 
 Note that these speculative models currently need to be run without tensor parallelism, although
 it is possible to run the main model using tensor parallelism (see example above). Since the
@@ -177,31 +185,33 @@ A variety of speculative models of this type are available on HF hub:
 The following code configures vLLM to use speculative decoding where proposals are generated by
 an [EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency)](https://arxiv.org/pdf/2401.15077) based draft model. A more detailed example for offline mode, including how to extract request level acceptance rate, can be found [here](gh-file:examples/offline_inference/eagle.py).
 
-```python
+??? Code
-from vllm import LLM, SamplingParams
 
-prompts = [
+    ```python
+    from vllm import LLM, SamplingParams
+
+    prompts = [
         "The future of AI is",
-]
+    ]
-sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
+    sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
 
-llm = LLM(
+    llm = LLM(
         model="meta-llama/Meta-Llama-3-8B-Instruct",
         tensor_parallel_size=4,
         speculative_config={
             "model": "yuhuili/EAGLE-LLaMA3-Instruct-8B",
             "draft_tensor_parallel_size": 1,
         },
-)
+    )
 
-outputs = llm.generate(prompts, sampling_params)
+    outputs = llm.generate(prompts, sampling_params)
 
-for output in outputs:
+    for output in outputs:
         prompt = output.prompt
         generated_text = output.outputs[0].text
         print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
 
-```
+    ```
 
 A few important things to consider when using the EAGLE based draft models:
 

docs/features/structured_outputs.md

@@ -33,28 +33,32 @@ text.
 
 Now let´s see an example for each of the cases, starting with the `guided_choice`, as it´s the easiest one:
 
-```python
+??? Code
-from openai import OpenAI
+
-client = OpenAI(
+    ```python
+    from openai import OpenAI
+    client = OpenAI(
         base_url="http://localhost:8000/v1",
         api_key="-",
-)
+    )
-model = client.models.list().data[0].id
+    model = client.models.list().data[0].id
 
-completion = client.chat.completions.create(
+    completion = client.chat.completions.create(
         model=model,
         messages=[
             {"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"}
         ],
         extra_body={"guided_choice": ["positive", "negative"]},
-)
+    )
-print(completion.choices[0].message.content)
+    print(completion.choices[0].message.content)
-```
+    ```
 
 The next example shows how to use the `guided_regex`. The idea is to generate an email address, given a simple regex template:
 
-```python
+??? Code
-completion = client.chat.completions.create(
+
+    ```python
+    completion = client.chat.completions.create(
         model=model,
         messages=[
             {
@@ -63,9 +67,9 @@ completion = client.chat.completions.create(
             }
         ],
         extra_body={"guided_regex": r"\w+@\w+\.com\n", "stop": ["\n"]},
-)
+    )
-print(completion.choices[0].message.content)
+    print(completion.choices[0].message.content)
-```
+    ```
 
 One of the most relevant features in structured text generation is the option to generate a valid JSON with pre-defined fields and formats.
 For this we can use the `guided_json` parameter in two different ways:
@@ -75,24 +79,26 @@ For this we can use the `guided_json` parameter in two different ways:
 
 The next example shows how to use the `guided_json` parameter with a Pydantic model:
 
-```python
+??? Code
-from pydantic import BaseModel
-from enum import Enum
 
-class CarType(str, Enum):
+    ```python
+    from pydantic import BaseModel
+    from enum import Enum
+
+    class CarType(str, Enum):
         sedan = "sedan"
         suv = "SUV"
         truck = "Truck"
         coupe = "Coupe"
 
-class CarDescription(BaseModel):
+    class CarDescription(BaseModel):
         brand: str
         model: str
         car_type: CarType
 
-json_schema = CarDescription.model_json_schema()
+    json_schema = CarDescription.model_json_schema()
 
-completion = client.chat.completions.create(
+    completion = client.chat.completions.create(
         model=model,
         messages=[
             {
@@ -107,9 +113,9 @@ completion = client.chat.completions.create(
                 "schema": CarDescription.model_json_schema()
             },
         },
-)
+    )
-print(completion.choices[0].message.content)
+    print(completion.choices[0].message.content)
-```
+    ```
 
 !!! tip
     While not strictly necessary, normally it´s better to indicate in the prompt the
@@ -121,8 +127,10 @@ difficult to use, but it´s really powerful. It allows us to define complete
 languages like SQL queries. It works by using a context free EBNF grammar.
 As an example, we can use to define a specific format of simplified SQL queries:
 
-```python
+??? Code
-simplified_sql_grammar = """
+
+    ```python
+    simplified_sql_grammar = """
         root ::= select_statement
 
         select_statement ::= "SELECT " column " from " table " where " condition
@@ -134,9 +142,9 @@ simplified_sql_grammar = """
         condition ::= column "= " number
 
         number ::= "1 " | "2 "
-"""
+    """
 
-completion = client.chat.completions.create(
+    completion = client.chat.completions.create(
         model=model,
         messages=[
             {
@@ -145,9 +153,9 @@ completion = client.chat.completions.create(
             }
         ],
         extra_body={"guided_grammar": simplified_sql_grammar},
-)
+    )
-print(completion.choices[0].message.content)
+    print(completion.choices[0].message.content)
-```
+    ```
 
 See also: [full example](https://docs.vllm.ai/en/latest/examples/online_serving/structured_outputs.html)
 
@@ -161,16 +169,18 @@ vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --reasoning-parser deepseek_r
 
 Note that you can use reasoning with any provided structured outputs feature. The following uses one with JSON schema:
 
-```python
+??? Code
-from pydantic import BaseModel
+
+    ```python
+    from pydantic import BaseModel
 
 
-class People(BaseModel):
+    class People(BaseModel):
         name: str
         age: int
 
 
-completion = client.chat.completions.create(
+    completion = client.chat.completions.create(
         model=model,
         messages=[
             {
@@ -185,10 +195,10 @@ completion = client.chat.completions.create(
                 "schema": People.model_json_schema()
             }
         },
-)
+    )
-print("reasoning_content: ", completion.choices[0].message.reasoning_content)
+    print("reasoning_content: ", completion.choices[0].message.reasoning_content)
-print("content: ", completion.choices[0].message.content)
+    print("content: ", completion.choices[0].message.content)
-```
+    ```
 
 See also: [full example](https://docs.vllm.ai/en/latest/examples/online_serving/structured_outputs.html)
 
@@ -202,33 +212,33 @@ For the following examples, vLLM was setup using `vllm serve meta-llama/Llama-3.
 
 Here is a simple example demonstrating how to get structured output using Pydantic models:
 
-```python
+??? Code
-from pydantic import BaseModel
-from openai import OpenAI
 
-class Info(BaseModel):
+    ```python
+    from pydantic import BaseModel
+    from openai import OpenAI
+
+    class Info(BaseModel):
         name: str
         age: int
 
-client = OpenAI(base_url="http://0.0.0.0:8000/v1", api_key="dummy")
+    client = OpenAI(base_url="http://0.0.0.0:8000/v1", api_key="dummy")
-model = client.models.list().data[0].id
+    model = client.models.list().data[0].id
-completion = client.beta.chat.completions.parse(
+    completion = client.beta.chat.completions.parse(
         model=model,
         messages=[
             {"role": "system", "content": "You are a helpful assistant."},
             {"role": "user", "content": "My name is Cameron, I'm 28. What's my name and age?"},
         ],
         response_format=Info,
-)
+    )
 
-message = completion.choices[0].message
+    message = completion.choices[0].message
-print(message)
+    print(message)
-assert message.parsed
+    assert message.parsed
-print("Name:", message.parsed.name)
+    print("Name:", message.parsed.name)
-print("Age:", message.parsed.age)
+    print("Age:", message.parsed.age)
-```
+    ```
-
-Output:
 
 ```console
 ParsedChatCompletionMessage[Testing](content='{"name": "Cameron", "age": 28}', refusal=None, role='assistant', audio=None, function_call=None, tool_calls=[], parsed=Testing(name='Cameron', age=28))
@@ -238,35 +248,37 @@ Age: 28
 
 Here is a more complex example using nested Pydantic models to handle a step-by-step math solution:
 
-```python
+??? Code
-from typing import List
-from pydantic import BaseModel
-from openai import OpenAI
 
-class Step(BaseModel):
+    ```python
+    from typing import List
+    from pydantic import BaseModel
+    from openai import OpenAI
+
+    class Step(BaseModel):
         explanation: str
         output: str
 
-class MathResponse(BaseModel):
+    class MathResponse(BaseModel):
         steps: list[Step]
         final_answer: str
 
-completion = client.beta.chat.completions.parse(
+    completion = client.beta.chat.completions.parse(
         model=model,
         messages=[
             {"role": "system", "content": "You are a helpful expert math tutor."},
             {"role": "user", "content": "Solve 8x + 31 = 2."},
         ],
         response_format=MathResponse,
-)
+    )
 
-message = completion.choices[0].message
+    message = completion.choices[0].message
-print(message)
+    print(message)
-assert message.parsed
+    assert message.parsed
-for i, step in enumerate(message.parsed.steps):
+    for i, step in enumerate(message.parsed.steps):
         print(f"Step #{i}:", step)
-print("Answer:", message.parsed.final_answer)
+    print("Answer:", message.parsed.final_answer)
-```
+    ```
 
 Output:
 
@@ -296,19 +308,21 @@ These parameters can be used in the same way as the parameters from the Online
 Serving examples above. One example for the usage of the `choice` parameter is
 shown below:
 
-```python
+??? Code
-from vllm import LLM, SamplingParams
-from vllm.sampling_params import GuidedDecodingParams
 
-llm = LLM(model="HuggingFaceTB/SmolLM2-1.7B-Instruct")
+    ```python
+    from vllm import LLM, SamplingParams
+    from vllm.sampling_params import GuidedDecodingParams
 
-guided_decoding_params = GuidedDecodingParams(choice=["Positive", "Negative"])
+    llm = LLM(model="HuggingFaceTB/SmolLM2-1.7B-Instruct")
-sampling_params = SamplingParams(guided_decoding=guided_decoding_params)
+
-outputs = llm.generate(
+    guided_decoding_params = GuidedDecodingParams(choice=["Positive", "Negative"])
+    sampling_params = SamplingParams(guided_decoding=guided_decoding_params)
+    outputs = llm.generate(
         prompts="Classify this sentiment: vLLM is wonderful!",
         sampling_params=sampling_params,
-)
+    )
-print(outputs[0].outputs[0].text)
+    print(outputs[0].outputs[0].text)
-```
+    ```
 
 See also: [full example](https://docs.vllm.ai/en/latest/examples/online_serving/structured_outputs.html)

docs/features/tool_calling.md

@@ -15,17 +15,19 @@ vllm serve meta-llama/Llama-3.1-8B-Instruct \
 
 Next, make a request to the model that should result in it using the available tools:
 
-```python
+??? Code
-from openai import OpenAI
-import json
 
-client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")
+    ```python
+    from openai import OpenAI
+    import json
 
-def get_weather(location: str, unit: str):
+    client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")
+
+    def get_weather(location: str, unit: str):
         return f"Getting the weather for {location} in {unit}..."
-tool_functions = {"get_weather": get_weather}
+    tool_functions = {"get_weather": get_weather}
 
-tools = [{
+    tools = [{
         "type": "function",
         "function": {
             "name": "get_weather",
@@ -39,20 +41,20 @@ tools = [{
                 "required": ["location", "unit"]
             }
         }
-}]
+    }]
 
-response = client.chat.completions.create(
+    response = client.chat.completions.create(
         model=client.models.list().data[0].id,
         messages=[{"role": "user", "content": "What's the weather like in San Francisco?"}],
         tools=tools,
         tool_choice="auto"
-)
+    )
 
-tool_call = response.choices[0].message.tool_calls[0].function
+    tool_call = response.choices[0].message.tool_calls[0].function
-print(f"Function called: {tool_call.name}")
+    print(f"Function called: {tool_call.name}")
-print(f"Arguments: {tool_call.arguments}")
+    print(f"Arguments: {tool_call.arguments}")
-print(f"Result: {get_weather(**json.loads(tool_call.arguments))}")
+    print(f"Result: {get_weather(**json.loads(tool_call.arguments))}")
-```
+    ```
 
 Example output:
 
@@ -301,16 +303,18 @@ A tool parser plugin is a Python file containing one or more ToolParser implemen
 
 Here is a summary of a plugin file:
 
-```python
+??? Code
 
-# import the required packages
+    ```python
 
-# define a tool parser and register it to vllm
+    # import the required packages
-# the name list in register_module can be used
+
-# in --tool-call-parser. you can define as many
+    # define a tool parser and register it to vllm
-# tool parsers as you want here.
+    # the name list in register_module can be used
-@ToolParserManager.register_module(["example"])
+    # in --tool-call-parser. you can define as many
-class ExampleToolParser(ToolParser):
+    # tool parsers as you want here.
+    @ToolParserManager.register_module(["example"])
+    class ExampleToolParser(ToolParser):
         def __init__(self, tokenizer: AnyTokenizer):
             super().__init__(tokenizer)
 
@@ -343,7 +347,7 @@ class ExampleToolParser(ToolParser):
                                                 tool_calls=[],
                                                 content=text)
 
-```
+    ```
 
 Then you can use this plugin in the command line like this.
 

docs/getting_started/installation/cpu.md

@@ -76,11 +76,13 @@ Currently, there are no pre-built CPU wheels.
 
 ### Build image from source
 
-```console
+??? Commands
-$ docker build -f docker/Dockerfile.cpu --tag vllm-cpu-env --target vllm-openai .
 
-# Launching OpenAI server 
+    ```console
-$ docker run --rm \
+    $ docker build -f docker/Dockerfile.cpu --tag vllm-cpu-env --target vllm-openai .
+
+    # Launching OpenAI server 
+    $ docker run --rm \
                 --privileged=true \
                 --shm-size=4g \
                 -p 8000:8000 \
@@ -90,7 +92,7 @@ $ docker run --rm \
                 --model=meta-llama/Llama-3.2-1B-Instruct \
                 --dtype=bfloat16 \
                 other vLLM OpenAI server arguments
-```
+    ```
 
 !!! tip
     For ARM or Apple silicon, use `docker/Dockerfile.arm`
@@ -144,32 +146,34 @@ vllm serve facebook/opt-125m
 
 - If using vLLM CPU backend on a machine with hyper-threading, it is recommended to bind only one OpenMP thread on each physical CPU core using `VLLM_CPU_OMP_THREADS_BIND` or using auto thread binding feature by default. On a hyper-threading enabled platform with 16 logical CPU cores / 8 physical CPU cores:
 
-```console
+??? Commands
-$ lscpu -e # check the mapping between logical CPU cores and physical CPU cores
 
-# The "CPU" column means the logical CPU core IDs, and the "CORE" column means the physical core IDs. On this platform, two logical cores are sharing one physical core.
+    ```console
-CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE    MAXMHZ   MINMHZ      MHZ
+    $ lscpu -e # check the mapping between logical CPU cores and physical CPU cores
-0    0      0    0 0:0:0:0          yes 2401.0000 800.0000  800.000
-1    0      0    1 1:1:1:0          yes 2401.0000 800.0000  800.000
-2    0      0    2 2:2:2:0          yes 2401.0000 800.0000  800.000
-3    0      0    3 3:3:3:0          yes 2401.0000 800.0000  800.000
-4    0      0    4 4:4:4:0          yes 2401.0000 800.0000  800.000
-5    0      0    5 5:5:5:0          yes 2401.0000 800.0000  800.000
-6    0      0    6 6:6:6:0          yes 2401.0000 800.0000  800.000
-7    0      0    7 7:7:7:0          yes 2401.0000 800.0000  800.000
-8    0      0    0 0:0:0:0          yes 2401.0000 800.0000  800.000
-9    0      0    1 1:1:1:0          yes 2401.0000 800.0000  800.000
-10   0      0    2 2:2:2:0          yes 2401.0000 800.0000  800.000
-11   0      0    3 3:3:3:0          yes 2401.0000 800.0000  800.000
-12   0      0    4 4:4:4:0          yes 2401.0000 800.0000  800.000
-13   0      0    5 5:5:5:0          yes 2401.0000 800.0000  800.000
-14   0      0    6 6:6:6:0          yes 2401.0000 800.0000  800.000
-15   0      0    7 7:7:7:0          yes 2401.0000 800.0000  800.000
 
-# On this platform, it is recommend to only bind openMP threads on logical CPU cores 0-7 or 8-15
+    # The "CPU" column means the logical CPU core IDs, and the "CORE" column means the physical core IDs. On this platform, two logical cores are sharing one physical core.
-$ export VLLM_CPU_OMP_THREADS_BIND=0-7
+    CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE    MAXMHZ   MINMHZ      MHZ
-$ python examples/offline_inference/basic/basic.py
+    0    0      0    0 0:0:0:0          yes 2401.0000 800.0000  800.000
-```
+    1    0      0    1 1:1:1:0          yes 2401.0000 800.0000  800.000
+    2    0      0    2 2:2:2:0          yes 2401.0000 800.0000  800.000
+    3    0      0    3 3:3:3:0          yes 2401.0000 800.0000  800.000
+    4    0      0    4 4:4:4:0          yes 2401.0000 800.0000  800.000
+    5    0      0    5 5:5:5:0          yes 2401.0000 800.0000  800.000
+    6    0      0    6 6:6:6:0          yes 2401.0000 800.0000  800.000
+    7    0      0    7 7:7:7:0          yes 2401.0000 800.0000  800.000
+    8    0      0    0 0:0:0:0          yes 2401.0000 800.0000  800.000
+    9    0      0    1 1:1:1:0          yes 2401.0000 800.0000  800.000
+    10   0      0    2 2:2:2:0          yes 2401.0000 800.0000  800.000
+    11   0      0    3 3:3:3:0          yes 2401.0000 800.0000  800.000
+    12   0      0    4 4:4:4:0          yes 2401.0000 800.0000  800.000
+    13   0      0    5 5:5:5:0          yes 2401.0000 800.0000  800.000
+    14   0      0    6 6:6:6:0          yes 2401.0000 800.0000  800.000
+    15   0      0    7 7:7:7:0          yes 2401.0000 800.0000  800.000
+
+    # On this platform, it is recommend to only bind openMP threads on logical CPU cores 0-7 or 8-15
+    $ export VLLM_CPU_OMP_THREADS_BIND=0-7
+    $ python examples/offline_inference/basic/basic.py
+    ```
 
 - If using vLLM CPU backend on a multi-socket machine with NUMA, be aware to set CPU cores using `VLLM_CPU_OMP_THREADS_BIND` to avoid cross NUMA node memory access.
 

docs/getting_started/installation/gpu/rocm.inc.md

@@ -90,6 +90,8 @@ Currently, there are no pre-built ROCm wheels.
 
 4. Build vLLM. For example, vLLM on ROCM 6.3 can be built with the following steps:
 
+    ??? Commands
+
         ```bash
         pip install --upgrade pip
 
@@ -201,8 +203,10 @@ DOCKER_BUILDKIT=1 docker build \
 
 To run the above docker image `vllm-rocm`, use the below command:
 
-```console
+??? Command
-docker run -it \
+
+    ```console
+    docker run -it \
     --network=host \
     --group-add=video \
     --ipc=host \
@@ -213,7 +217,7 @@ docker run -it \
     -v <path/to/model>:/app/model \
     vllm-rocm \
     bash
-```
+    ```
 
 Where the `<path/to/model>` is the location where the model is stored, for example, the weights for llama2 or llama3 models.
 

docs/getting_started/installation/intel_gaudi.md

@@ -200,7 +200,7 @@ INFO 08-01 21:37:59 hpu_model_runner.py:509] Generated 48 decode buckets: [(1, 1
 
 `min` determines the lowest value of the bucket. `step` determines the interval between buckets, and `max` determines the upper bound of the bucket. Furthermore, interval between `min` and `step` has special handling -- `min` gets multiplied by consecutive powers of two, until `step` gets reached. We call this the ramp-up phase and it is used for handling lower batch sizes with minimum wastage, while allowing larger padding on larger batch sizes.
 
-Example (with ramp-up)
+Example (with ramp-up):
 
 ```text
 min = 2, step = 32, max = 64
@@ -209,7 +209,7 @@ min = 2, step = 32, max = 64
 => buckets = ramp_up + stable => (2, 4, 8, 16, 32, 64)
 ```
 
-Example (without ramp-up)
+Example (without ramp-up):
 
 ```text
 min = 128, step = 128, max = 512
@@ -232,19 +232,21 @@ As an example, if a request of 3 sequences, with max sequence length of 412 come
 
 Warmup is an optional, but highly recommended step occurring before vLLM server starts listening. It executes a forward pass for each bucket with dummy data. The goal is to pre-compile all graphs and not incur any graph compilation overheads within bucket boundaries during server runtime. Each warmup step is logged during vLLM startup:
 
-```text
+??? Logs
-INFO 08-01 22:26:47 hpu_model_runner.py:1066] [Warmup][Prompt][1/24] batch_size:4 seq_len:1024 free_mem:79.16 GiB
+
-INFO 08-01 22:26:47 hpu_model_runner.py:1066] [Warmup][Prompt][2/24] batch_size:4 seq_len:896 free_mem:55.43 GiB
+    ```text
-INFO 08-01 22:26:48 hpu_model_runner.py:1066] [Warmup][Prompt][3/24] batch_size:4 seq_len:768 free_mem:55.43 GiB
+    INFO 08-01 22:26:47 hpu_model_runner.py:1066] [Warmup][Prompt][1/24] batch_size:4 seq_len:1024 free_mem:79.16 GiB
-...
+    INFO 08-01 22:26:47 hpu_model_runner.py:1066] [Warmup][Prompt][2/24] batch_size:4 seq_len:896 free_mem:55.43 GiB
-INFO 08-01 22:26:59 hpu_model_runner.py:1066] [Warmup][Prompt][24/24] batch_size:1 seq_len:128 free_mem:55.43 GiB
+    INFO 08-01 22:26:48 hpu_model_runner.py:1066] [Warmup][Prompt][3/24] batch_size:4 seq_len:768 free_mem:55.43 GiB
-INFO 08-01 22:27:00 hpu_model_runner.py:1066] [Warmup][Decode][1/48] batch_size:4 seq_len:2048 free_mem:55.43 GiB
+    ...
-INFO 08-01 22:27:00 hpu_model_runner.py:1066] [Warmup][Decode][2/48] batch_size:4 seq_len:1920 free_mem:55.43 GiB
+    INFO 08-01 22:26:59 hpu_model_runner.py:1066] [Warmup][Prompt][24/24] batch_size:1 seq_len:128 free_mem:55.43 GiB
-INFO 08-01 22:27:01 hpu_model_runner.py:1066] [Warmup][Decode][3/48] batch_size:4 seq_len:1792 free_mem:55.43 GiB
+    INFO 08-01 22:27:00 hpu_model_runner.py:1066] [Warmup][Decode][1/48] batch_size:4 seq_len:2048 free_mem:55.43 GiB
-...
+    INFO 08-01 22:27:00 hpu_model_runner.py:1066] [Warmup][Decode][2/48] batch_size:4 seq_len:1920 free_mem:55.43 GiB
-INFO 08-01 22:27:16 hpu_model_runner.py:1066] [Warmup][Decode][47/48] batch_size:2 seq_len:128 free_mem:55.43 GiB
+    INFO 08-01 22:27:01 hpu_model_runner.py:1066] [Warmup][Decode][3/48] batch_size:4 seq_len:1792 free_mem:55.43 GiB
-INFO 08-01 22:27:16 hpu_model_runner.py:1066] [Warmup][Decode][48/48] batch_size:1 seq_len:128 free_mem:55.43 GiB
+    ...
-```
+    INFO 08-01 22:27:16 hpu_model_runner.py:1066] [Warmup][Decode][47/48] batch_size:2 seq_len:128 free_mem:55.43 GiB
+    INFO 08-01 22:27:16 hpu_model_runner.py:1066] [Warmup][Decode][48/48] batch_size:1 seq_len:128 free_mem:55.43 GiB
+    ```
 
 This example uses the same buckets as in the [Bucketing Mechanism][gaudi-bucketing-mechanism] section. Each output line corresponds to execution of a single bucket. When bucket is executed for the first time, its graph is compiled and can be reused later on, skipping further graph compilations.
 
@@ -279,37 +281,39 @@ When there's large amount of requests pending, vLLM scheduler will attempt to fi
 
 Each described step is logged by vLLM server, as follows (negative values correspond to memory being released):
 
-```text
+??? Logs
-INFO 08-02 17:37:44 hpu_model_runner.py:493] Prompt bucket config (min, step, max_warmup) bs:[1, 32, 4], seq:[128, 128, 1024]
+
-INFO 08-02 17:37:44 hpu_model_runner.py:499] Generated 24 prompt buckets: [(1, 128), (1, 256), (1, 384), (1, 512), (1, 640), (1, 768), (1, 896), (1, 1024), (2, 128), (2, 256), (2, 384), (2, 512), (2, 640), (2, 768), (2, 896), (2, 1024), (4, 128), (4, 256), (4, 384), (4, 512), (4, 640), (4, 768), (4, 896), (4, 1024)]
+    ```text
-INFO 08-02 17:37:44 hpu_model_runner.py:504] Decode bucket config (min, step, max_warmup) bs:[1, 128, 4], seq:[128, 128, 2048]
+    INFO 08-02 17:37:44 hpu_model_runner.py:493] Prompt bucket config (min, step, max_warmup) bs:[1, 32, 4], seq:[128, 128, 1024]
-INFO 08-02 17:37:44 hpu_model_runner.py:509] Generated 48 decode buckets: [(1, 128), (1, 256), (1, 384), (1, 512), (1, 640), (1, 768), (1, 896), (1, 1024), (1, 1152), (1, 1280), (1, 1408), (1, 1536), (1, 1664), (1, 1792), (1, 1920), (1, 2048), (2, 128), (2, 256), (2, 384), (2, 512), (2, 640), (2, 768), (2, 896), (2, 1024), (2, 1152), (2, 1280), (2, 1408), (2, 1536), (2, 1664), (2, 1792), (2, 1920), (2, 2048), (4, 128), (4, 256), (4, 384), (4, 512), (4, 640), (4, 768), (4, 896), (4, 1024), (4, 1152), (4, 1280), (4, 1408), (4, 1536), (4, 1664), (4, 1792), (4, 1920), (4, 2048)]
+    INFO 08-02 17:37:44 hpu_model_runner.py:499] Generated 24 prompt buckets: [(1, 128), (1, 256), (1, 384), (1, 512), (1, 640), (1, 768), (1, 896), (1, 1024), (2, 128), (2, 256), (2, 384), (2, 512), (2, 640), (2, 768), (2, 896), (2, 1024), (4, 128), (4, 256), (4, 384), (4, 512), (4, 640), (4, 768), (4, 896), (4, 1024)]
-INFO 08-02 17:37:52 hpu_model_runner.py:430] Pre-loading model weights on hpu:0 took 14.97 GiB of device memory (14.97 GiB/94.62 GiB used) and 2.95 GiB of host memory (475.2 GiB/1007 GiB used)
+    INFO 08-02 17:37:44 hpu_model_runner.py:504] Decode bucket config (min, step, max_warmup) bs:[1, 128, 4], seq:[128, 128, 2048]
-INFO 08-02 17:37:52 hpu_model_runner.py:438] Wrapping in HPU Graph took 0 B of device memory (14.97 GiB/94.62 GiB used) and -252 KiB of host memory (475.2 GiB/1007 GiB used)
+    INFO 08-02 17:37:44 hpu_model_runner.py:509] Generated 48 decode buckets: [(1, 128), (1, 256), (1, 384), (1, 512), (1, 640), (1, 768), (1, 896), (1, 1024), (1, 1152), (1, 1280), (1, 1408), (1, 1536), (1, 1664), (1, 1792), (1, 1920), (1, 2048), (2, 128), (2, 256), (2, 384), (2, 512), (2, 640), (2, 768), (2, 896), (2, 1024), (2, 1152), (2, 1280), (2, 1408), (2, 1536), (2, 1664), (2, 1792), (2, 1920), (2, 2048), (4, 128), (4, 256), (4, 384), (4, 512), (4, 640), (4, 768), (4, 896), (4, 1024), (4, 1152), (4, 1280), (4, 1408), (4, 1536), (4, 1664), (4, 1792), (4, 1920), (4, 2048)]
-INFO 08-02 17:37:52 hpu_model_runner.py:442] Loading model weights took in total 14.97 GiB of device memory (14.97 GiB/94.62 GiB used) and 2.95 GiB of host memory (475.2 GiB/1007 GiB used)
+    INFO 08-02 17:37:52 hpu_model_runner.py:430] Pre-loading model weights on hpu:0 took 14.97 GiB of device memory (14.97 GiB/94.62 GiB used) and 2.95 GiB of host memory (475.2 GiB/1007 GiB used)
-INFO 08-02 17:37:54 hpu_worker.py:134] Model profiling run took 504 MiB of device memory (15.46 GiB/94.62 GiB used) and 180.9 MiB of host memory (475.4 GiB/1007 GiB used)
+    INFO 08-02 17:37:52 hpu_model_runner.py:438] Wrapping in HPU Graph took 0 B of device memory (14.97 GiB/94.62 GiB used) and -252 KiB of host memory (475.2 GiB/1007 GiB used)
-INFO 08-02 17:37:54 hpu_worker.py:158] Free device memory: 79.16 GiB, 39.58 GiB usable (gpu_memory_utilization=0.5), 15.83 GiB reserved for HPUGraphs (VLLM_GRAPH_RESERVED_MEM=0.4), 23.75 GiB reserved for KV cache
+    INFO 08-02 17:37:52 hpu_model_runner.py:442] Loading model weights took in total 14.97 GiB of device memory (14.97 GiB/94.62 GiB used) and 2.95 GiB of host memory (475.2 GiB/1007 GiB used)
-INFO 08-02 17:37:54 hpu_executor.py:85] # HPU blocks: 1519, # CPU blocks: 0
+    INFO 08-02 17:37:54 hpu_worker.py:134] Model profiling run took 504 MiB of device memory (15.46 GiB/94.62 GiB used) and 180.9 MiB of host memory (475.4 GiB/1007 GiB used)
-INFO 08-02 17:37:54 hpu_worker.py:190] Initializing cache engine took 23.73 GiB of device memory (39.2 GiB/94.62 GiB used) and -1.238 MiB of host memory (475.4 GiB/1007 GiB used)
+    INFO 08-02 17:37:54 hpu_worker.py:158] Free device memory: 79.16 GiB, 39.58 GiB usable (gpu_memory_utilization=0.5), 15.83 GiB reserved for HPUGraphs (VLLM_GRAPH_RESERVED_MEM=0.4), 23.75 GiB reserved for KV cache
-INFO 08-02 17:37:54 hpu_model_runner.py:1066] [Warmup][Prompt][1/24] batch_size:4 seq_len:1024 free_mem:55.43 GiB
+    INFO 08-02 17:37:54 hpu_executor.py:85] # HPU blocks: 1519, # CPU blocks: 0
-...
+    INFO 08-02 17:37:54 hpu_worker.py:190] Initializing cache engine took 23.73 GiB of device memory (39.2 GiB/94.62 GiB used) and -1.238 MiB of host memory (475.4 GiB/1007 GiB used)
-INFO 08-02 17:38:22 hpu_model_runner.py:1066] [Warmup][Decode][48/48] batch_size:1 seq_len:128 free_mem:55.43 GiB
+    INFO 08-02 17:37:54 hpu_model_runner.py:1066] [Warmup][Prompt][1/24] batch_size:4 seq_len:1024 free_mem:55.43 GiB
-INFO 08-02 17:38:22 hpu_model_runner.py:1159] Using 15.85 GiB/55.43 GiB of free device memory for HPUGraphs, 7.923 GiB for prompt and 7.923 GiB for decode (VLLM_GRAPH_PROMPT_RATIO=0.3)
+    ...
-INFO 08-02 17:38:22 hpu_model_runner.py:1066] [Warmup][Graph/Prompt][1/24] batch_size:1 seq_len:128 free_mem:55.43 GiB
+    INFO 08-02 17:38:22 hpu_model_runner.py:1066] [Warmup][Decode][48/48] batch_size:1 seq_len:128 free_mem:55.43 GiB
-...
+    INFO 08-02 17:38:22 hpu_model_runner.py:1159] Using 15.85 GiB/55.43 GiB of free device memory for HPUGraphs, 7.923 GiB for prompt and 7.923 GiB for decode (VLLM_GRAPH_PROMPT_RATIO=0.3)
-INFO 08-02 17:38:26 hpu_model_runner.py:1066] [Warmup][Graph/Prompt][11/24] batch_size:1 seq_len:896 free_mem:48.77 GiB
+    INFO 08-02 17:38:22 hpu_model_runner.py:1066] [Warmup][Graph/Prompt][1/24] batch_size:1 seq_len:128 free_mem:55.43 GiB
-INFO 08-02 17:38:27 hpu_model_runner.py:1066] [Warmup][Graph/Decode][1/48] batch_size:4 seq_len:128 free_mem:47.51 GiB
+    ...
-...
+    INFO 08-02 17:38:26 hpu_model_runner.py:1066] [Warmup][Graph/Prompt][11/24] batch_size:1 seq_len:896 free_mem:48.77 GiB
-INFO 08-02 17:38:41 hpu_model_runner.py:1066] [Warmup][Graph/Decode][48/48] batch_size:1 seq_len:2048 free_mem:47.35 GiB
+    INFO 08-02 17:38:27 hpu_model_runner.py:1066] [Warmup][Graph/Decode][1/48] batch_size:4 seq_len:128 free_mem:47.51 GiB
-INFO 08-02 17:38:41 hpu_model_runner.py:1066] [Warmup][Graph/Prompt][12/24] batch_size:4 seq_len:256 free_mem:47.35 GiB
+    ...
-INFO 08-02 17:38:42 hpu_model_runner.py:1066] [Warmup][Graph/Prompt][13/24] batch_size:2 seq_len:512 free_mem:45.91 GiB
+    INFO 08-02 17:38:41 hpu_model_runner.py:1066] [Warmup][Graph/Decode][48/48] batch_size:1 seq_len:2048 free_mem:47.35 GiB
-INFO 08-02 17:38:42 hpu_model_runner.py:1066] [Warmup][Graph/Prompt][14/24] batch_size:1 seq_len:1024 free_mem:44.48 GiB
+    INFO 08-02 17:38:41 hpu_model_runner.py:1066] [Warmup][Graph/Prompt][12/24] batch_size:4 seq_len:256 free_mem:47.35 GiB
-INFO 08-02 17:38:43 hpu_model_runner.py:1066] [Warmup][Graph/Prompt][15/24] batch_size:2 seq_len:640 free_mem:43.03 GiB
+    INFO 08-02 17:38:42 hpu_model_runner.py:1066] [Warmup][Graph/Prompt][13/24] batch_size:2 seq_len:512 free_mem:45.91 GiB
-INFO 08-02 17:38:43 hpu_model_runner.py:1128] Graph/Prompt captured:15 (62.5%) used_mem:14.03 GiB buckets:[(1, 128), (1, 256), (1, 384), (1, 512), (1, 640), (1, 768), (1, 896), (1, 1024), (2, 128), (2, 256), (2, 384), (2, 512), (2, 640), (4, 128), (4, 256)]
+    INFO 08-02 17:38:42 hpu_model_runner.py:1066] [Warmup][Graph/Prompt][14/24] batch_size:1 seq_len:1024 free_mem:44.48 GiB
-INFO 08-02 17:38:43 hpu_model_runner.py:1128] Graph/Decode captured:48 (100.0%) used_mem:161.9 MiB buckets:[(1, 128), (1, 256), (1, 384), (1, 512), (1, 640), (1, 768), (1, 896), (1, 1024), (1, 1152), (1, 1280), (1, 1408), (1, 1536), (1, 1664), (1, 1792), (1, 1920), (1, 2048), (2, 128), (2, 256), (2, 384), (2, 512), (2, 640), (2, 768), (2, 896), (2, 1024), (2, 1152), (2, 1280), (2, 1408), (2, 1536), (2, 1664), (2, 1792), (2, 1920), (2, 2048), (4, 128), (4, 256), (4, 384), (4, 512), (4, 640), (4, 768), (4, 896), (4, 1024), (4, 1152), (4, 1280), (4, 1408), (4, 1536), (4, 1664), (4, 1792), (4, 1920), (4, 2048)]
+    INFO 08-02 17:38:43 hpu_model_runner.py:1066] [Warmup][Graph/Prompt][15/24] batch_size:2 seq_len:640 free_mem:43.03 GiB
-INFO 08-02 17:38:43 hpu_model_runner.py:1206] Warmup finished in 49 secs, allocated 14.19 GiB of device memory
+    INFO 08-02 17:38:43 hpu_model_runner.py:1128] Graph/Prompt captured:15 (62.5%) used_mem:14.03 GiB buckets:[(1, 128), (1, 256), (1, 384), (1, 512), (1, 640), (1, 768), (1, 896), (1, 1024), (2, 128), (2, 256), (2, 384), (2, 512), (2, 640), (4, 128), (4, 256)]
-INFO 08-02 17:38:43 hpu_executor.py:91] init_cache_engine took 37.92 GiB of device memory (53.39 GiB/94.62 GiB used) and 57.86 MiB of host memory (475.4 GiB/1007 GiB used)
+    INFO 08-02 17:38:43 hpu_model_runner.py:1128] Graph/Decode captured:48 (100.0%) used_mem:161.9 MiB buckets:[(1, 128), (1, 256), (1, 384), (1, 512), (1, 640), (1, 768), (1, 896), (1, 1024), (1, 1152), (1, 1280), (1, 1408), (1, 1536), (1, 1664), (1, 1792), (1, 1920), (1, 2048), (2, 128), (2, 256), (2, 384), (2, 512), (2, 640), (2, 768), (2, 896), (2, 1024), (2, 1152), (2, 1280), (2, 1408), (2, 1536), (2, 1664), (2, 1792), (2, 1920), (2, 2048), (4, 128), (4, 256), (4, 384), (4, 512), (4, 640), (4, 768), (4, 896), (4, 1024), (4, 1152), (4, 1280), (4, 1408), (4, 1536), (4, 1664), (4, 1792), (4, 1920), (4, 2048)]
-```
+    INFO 08-02 17:38:43 hpu_model_runner.py:1206] Warmup finished in 49 secs, allocated 14.19 GiB of device memory
+    INFO 08-02 17:38:43 hpu_executor.py:91] init_cache_engine took 37.92 GiB of device memory (53.39 GiB/94.62 GiB used) and 57.86 MiB of host memory (475.4 GiB/1007 GiB used)
+    ```
 
 ### Recommended vLLM Parameters
 

docs/getting_started/quickstart.md

@@ -147,20 +147,22 @@ curl http://localhost:8000/v1/completions \
 
 Since this server is compatible with OpenAI API, you can use it as a drop-in replacement for any applications using OpenAI API. For example, another way to query the server is via the `openai` Python package:
 
-```python
+??? Code
-from openai import OpenAI
 
-# Modify OpenAI's API key and API base to use vLLM's API server.
+    ```python
-openai_api_key = "EMPTY"
+    from openai import OpenAI
-openai_api_base = "http://localhost:8000/v1"
+
-client = OpenAI(
+    # Modify OpenAI's API key and API base to use vLLM's API server.
+    openai_api_key = "EMPTY"
+    openai_api_base = "http://localhost:8000/v1"
+    client = OpenAI(
         api_key=openai_api_key,
         base_url=openai_api_base,
-)
+    )
-completion = client.completions.create(model="Qwen/Qwen2.5-1.5B-Instruct",
+    completion = client.completions.create(model="Qwen/Qwen2.5-1.5B-Instruct",
                                         prompt="San Francisco is a")
-print("Completion result:", completion)
+    print("Completion result:", completion)
-```
+    ```
 
 A more detailed client example can be found here: <gh-file:examples/online_serving/openai_completion_client.py>
 
@@ -184,26 +186,28 @@ curl http://localhost:8000/v1/chat/completions \
 
 Alternatively, you can use the `openai` Python package:
 
-```python
+??? Code
-from openai import OpenAI
-# Set OpenAI's API key and API base to use vLLM's API server.
-openai_api_key = "EMPTY"
-openai_api_base = "http://localhost:8000/v1"
 
-client = OpenAI(
+    ```python
+    from openai import OpenAI
+    # Set OpenAI's API key and API base to use vLLM's API server.
+    openai_api_key = "EMPTY"
+    openai_api_base = "http://localhost:8000/v1"
+
+    client = OpenAI(
         api_key=openai_api_key,
         base_url=openai_api_base,
-)
+    )
 
-chat_response = client.chat.completions.create(
+    chat_response = client.chat.completions.create(
         model="Qwen/Qwen2.5-1.5B-Instruct",
         messages=[
             {"role": "system", "content": "You are a helpful assistant."},
             {"role": "user", "content": "Tell me a joke."},
         ]
-)
+    )
-print("Chat response:", chat_response)
+    print("Chat response:", chat_response)
-```
+    ```
 
 ## On Attention Backends
 

docs/models/generative_models.md

@@ -85,11 +85,13 @@ and automatically applies the model's [chat template](https://huggingface.co/doc
     In general, only instruction-tuned models have a chat template.
     Base models may perform poorly as they are not trained to respond to the chat conversation.
 
-```python
+??? Code
-from vllm import LLM
 
-llm = LLM(model="meta-llama/Meta-Llama-3-8B-Instruct")
+    ```python
-conversation = [
+    from vllm import LLM
+
+    llm = LLM(model="meta-llama/Meta-Llama-3-8B-Instruct")
+    conversation = [
         {
             "role": "system",
             "content": "You are a helpful assistant"
@@ -106,14 +108,14 @@ conversation = [
             "role": "user",
             "content": "Write an essay about the importance of higher education.",
         },
-]
+    ]
-outputs = llm.chat(conversation)
+    outputs = llm.chat(conversation)
 
-for output in outputs:
+    for output in outputs:
         prompt = output.prompt
         generated_text = output.outputs[0].text
         print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
-```
+    ```
 
 A code example can be found here: <gh-file:examples/offline_inference/basic/chat.py>
 

docs/models/supported_models.md

@@ -70,7 +70,10 @@ To make your model compatible with the Transformers backend, it needs:
 2. `MyAttention` must use `ALL_ATTENTION_FUNCTIONS` to call attention.
 3. `MyModel` must contain `_supports_attention_backend = True`.
 
-```python title="modeling_my_model.py"
+<details>
+<summary>modeling_my_model.py</summary>
+
+```python
 
 from transformers import PreTrainedModel
 from torch import nn
@@ -93,6 +96,8 @@ class MyModel(PreTrainedModel):
     _supports_attention_backend = True
 ```
 
+</details>
+
 Here is what happens in the background when this model is loaded:
 
 1. The config is loaded.
@@ -103,7 +108,10 @@ That's it!
 
 For your model to be compatible with vLLM's tensor parallel and/or pipeline parallel features, you must add `base_model_tp_plan` and/or `base_model_pp_plan` to your model's config class:
 
-```python title="configuration_my_model.py"
+<details>
+<summary>configuration_my_model.py</summary>
+
+```python
 
 from transformers import PretrainedConfig
 
@@ -123,6 +131,8 @@ class MyConfig(PretrainedConfig):
     }
 ```
 
+</details>
+
 - `base_model_tp_plan` is a `dict` that maps fully qualified layer name patterns to tensor parallel styles (currently only `"colwise"` and `"rowwise"` are supported).
 - `base_model_pp_plan` is a `dict` that maps direct child layer names to `tuple`s of `list`s of `str`s:
     * You only need to do this for layers which are not present on all pipeline stages
@@ -198,6 +208,9 @@ huggingface-cli scan-cache --dir ~/.cache/huggingface/hub
 
 Use the Hugging Face CLI to interactively [delete downloaded model](https://huggingface.co/docs/huggingface_hub/guides/manage-cache#clean-your-cache) from the cache:
 
+<details>
+<summary>Commands</summary>
+
 ```console
 # The `delete-cache` command requires extra dependencies to work with the TUI.
 # Please run `pip install huggingface_hub[cli]` to install them.
@@ -224,6 +237,8 @@ Start deletion.
 Done. Deleted 1 repo(s) and 0 revision(s) for a total of 438.9M.
 ```
 
+</details>
+
 #### Using a proxy
 
 Here are some tips for loading/downloading models from Hugging Face using a proxy:
@@ -601,6 +616,8 @@ Specified using `--task generate`.
 
     For the best results, we recommend using the following dependency versions (tested on A10 and L40):
 
+    ??? Dependency versions
+
         ```text
         # Core vLLM-compatible dependencies with Molmo accuracy setup (tested on L40)
         torch==2.5.1

docs/serving/integrations/langchain.md

@@ -13,19 +13,21 @@ pip install langchain langchain_community -q
 
 To run inference on a single or multiple GPUs, use `VLLM` class from `langchain`.
 
-```python
+??? Code
-from langchain_community.llms import VLLM
 
-llm = VLLM(model="mosaicml/mpt-7b",
+    ```python
+    from langchain_community.llms import VLLM
+
+    llm = VLLM(model="mosaicml/mpt-7b",
             trust_remote_code=True,  # mandatory for hf models
             max_new_tokens=128,
             top_k=10,
             top_p=0.95,
             temperature=0.8,
             # tensor_parallel_size=... # for distributed inference
-)
+    )
 
-print(llm("What is the capital of France ?"))
+    print(llm("What is the capital of France ?"))
-```
+    ```
 
 Please refer to this [Tutorial](https://python.langchain.com/docs/integrations/llms/vllm) for more details.

docs/serving/openai_compatible_server.md

@@ -15,22 +15,24 @@ vllm serve NousResearch/Meta-Llama-3-8B-Instruct \
 
 To call the server, in your preferred text editor, create a script that uses an HTTP client. Include any messages that you want to send to the model. Then run that script. Below is an example script using the [official OpenAI Python client](https://github.com/openai/openai-python).
 
-```python
+??? Code
-from openai import OpenAI
+
-client = OpenAI(
+    ```python
+    from openai import OpenAI
+    client = OpenAI(
         base_url="http://localhost:8000/v1",
         api_key="token-abc123",
-)
+    )
 
-completion = client.chat.completions.create(
+    completion = client.chat.completions.create(
         model="NousResearch/Meta-Llama-3-8B-Instruct",
         messages=[
             {"role": "user", "content": "Hello!"}
         ]
-)
+    )
 
-print(completion.choices[0].message)
+    print(completion.choices[0].message)
-```
+    ```
 
 !!! tip
     vLLM supports some parameters that are not supported by OpenAI, `top_k` for example.
@@ -147,8 +149,10 @@ with `--enable-request-id-headers`.
 > rather than within the vLLM layer for this reason.
 > See [this PR](https://github.com/vllm-project/vllm/pull/11529) for more details.
 
-```python
+??? Code
-completion = client.chat.completions.create(
+
+    ```python
+    completion = client.chat.completions.create(
         model="NousResearch/Meta-Llama-3-8B-Instruct",
         messages=[
             {"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"}
@@ -156,18 +160,18 @@ completion = client.chat.completions.create(
         extra_headers={
             "x-request-id": "sentiment-classification-00001",
         }
-)
+    )
-print(completion._request_id)
+    print(completion._request_id)
 
-completion = client.completions.create(
+    completion = client.completions.create(
         model="NousResearch/Meta-Llama-3-8B-Instruct",
         prompt="A robot may not injure a human being",
         extra_headers={
             "x-request-id": "completion-test",
         }
-)
+    )
-print(completion._request_id)
+    print(completion._request_id)
-```
+    ```
 
 ## API Reference
 
@@ -184,15 +188,19 @@ Code example: <gh-file:examples/online_serving/openai_completion_client.py>
 
 The following [sampling parameters][sampling-params] are supported.
 
-```python
+??? Code
---8<-- "vllm/entrypoints/openai/protocol.py:completion-sampling-params"
+
-```
+    ```python
+    --8<-- "vllm/entrypoints/openai/protocol.py:completion-sampling-params"
+    ```
 
 The following extra parameters are supported:
 
-```python
+??? Code
---8<-- "vllm/entrypoints/openai/protocol.py:completion-extra-params"
+
-```
+    ```python
+    --8<-- "vllm/entrypoints/openai/protocol.py:completion-extra-params"
+    ```
 
 [](){ #chat-api }
 
@@ -212,15 +220,19 @@ Code example: <gh-file:examples/online_serving/openai_chat_completion_client.py>
 
 The following [sampling parameters][sampling-params] are supported.
 
-```python
+??? Code
---8<-- "vllm/entrypoints/openai/protocol.py:chat-completion-sampling-params"
+
-```
+    ```python
+    --8<-- "vllm/entrypoints/openai/protocol.py:chat-completion-sampling-params"
+    ```
 
 The following extra parameters are supported:
 
-```python
+??? Code
---8<-- "vllm/entrypoints/openai/protocol.py:chat-completion-extra-params"
+
-```
+    ```python
+    --8<-- "vllm/entrypoints/openai/protocol.py:chat-completion-extra-params"
+    ```
 
 [](){ #embeddings-api }
 
@@ -259,6 +271,8 @@ and passing a list of `messages` in the request. Refer to the examples below for
 
     Since the request schema is not defined by OpenAI client, we post a request to the server using the lower-level `requests` library:
 
+    ??? Code
+
         ```python
         import requests
 
@@ -316,15 +330,19 @@ The following [pooling parameters][pooling-params] are supported.
 
 The following extra parameters are supported by default:
 
-```python
+??? Code
---8<-- "vllm/entrypoints/openai/protocol.py:embedding-extra-params"
+
-```
+    ```python
+    --8<-- "vllm/entrypoints/openai/protocol.py:embedding-extra-params"
+    ```
 
 For chat-like input (i.e. if `messages` is passed), these extra parameters are supported instead:
 
-```python
+??? Code
---8<-- "vllm/entrypoints/openai/protocol.py:chat-embedding-extra-params"
+
-```
+    ```python
+    --8<-- "vllm/entrypoints/openai/protocol.py:chat-embedding-extra-params"
+    ```
 
 [](){ #transcriptions-api }
 
@@ -343,15 +361,19 @@ Code example: <gh-file:examples/online_serving/openai_transcription_client.py>
 
 The following [sampling parameters][sampling-params] are supported.
 
-```python
+??? Code
---8<-- "vllm/entrypoints/openai/protocol.py:transcription-sampling-params"
+
-```
+    ```python
+    --8<-- "vllm/entrypoints/openai/protocol.py:transcription-sampling-params"
+    ```
 
 The following extra parameters are supported:
 
-```python
+??? Code
---8<-- "vllm/entrypoints/openai/protocol.py:transcription-extra-params"
+
-```
+    ```python
+    --8<-- "vllm/entrypoints/openai/protocol.py:transcription-extra-params"
+    ```
 
 [](){ #tokenizer-api }
 
@@ -387,8 +409,6 @@ Code example: <gh-file:examples/online_serving/openai_classification_client.py>
 
 You can classify multiple texts by passing an array of strings:
 
-Request:
-
 ```bash
 curl -v "http://127.0.0.1:8000/classify" \
   -H "Content-Type: application/json" \
@@ -401,10 +421,10 @@ curl -v "http://127.0.0.1:8000/classify" \
   }'
 ```
 
-Response:
+??? Response
 
-```bash
+    ```bash
-{
+    {
       "id": "classify-7c87cac407b749a6935d8c7ce2a8fba2",
       "object": "list",
       "created": 1745383065,
@@ -435,13 +455,11 @@ Response:
         "completion_tokens": 0,
         "prompt_tokens_details": null
       }
-}
+    }
-```
+    ```
 
 You can also pass a string directly to the `input` field:
 
-Request:
-
 ```bash
 curl -v "http://127.0.0.1:8000/classify" \
   -H "Content-Type: application/json" \
@@ -451,10 +469,10 @@ curl -v "http://127.0.0.1:8000/classify" \
   }'
 ```
 
-Response:
+??? Response
 
-```bash
+    ```bash
-{
+    {
       "id": "classify-9bf17f2847b046c7b2d5495f4b4f9682",
       "object": "list",
       "created": 1745383213,
@@ -476,8 +494,8 @@ Response:
         "completion_tokens": 0,
         "prompt_tokens_details": null
       }
-}
+    }
-```
+    ```
 
 #### Extra parameters
 
@@ -508,8 +526,6 @@ Code example: <gh-file:examples/online_serving/openai_cross_encoder_score.py>
 
 You can pass a string to both `text_1` and `text_2`, forming a single sentence pair.
 
-Request:
-
 ```bash
 curl -X 'POST' \
   'http://127.0.0.1:8000/score' \
@@ -523,10 +539,10 @@ curl -X 'POST' \
 }'
 ```
 
-Response:
+??? Response
 
-```bash
+    ```bash
-{
+    {
       "id": "score-request-id",
       "object": "list",
       "created": 693447,
@@ -539,8 +555,8 @@ Response:
         }
       ],
       "usage": {}
-}
+    }
-```
+    ```
 
 #### Batch inference
 
@@ -548,10 +564,10 @@ You can pass a string to `text_1` and a list to `text_2`, forming multiple sente
 where each pair is built from `text_1` and a string in `text_2`.
 The total number of pairs is `len(text_2)`.
 
-Request:
+??? Request
 
-```bash
+    ```bash
-curl -X 'POST' \
+    curl -X 'POST' \
       'http://127.0.0.1:8000/score' \
       -H 'accept: application/json' \
       -H 'Content-Type: application/json' \
@@ -562,13 +578,13 @@ curl -X 'POST' \
         "The capital of Brazil is Brasilia.",
         "The capital of France is Paris."
       ]
-}'
+    }'
-```
+    ```
 
-Response:
+??? Response
 
-```bash
+    ```bash
-{
+    {
       "id": "score-request-id",
       "object": "list",
       "created": 693570,
@@ -586,17 +602,17 @@ Response:
         }
       ],
       "usage": {}
-}
+    }
-```
+    ```
 
 You can pass a list to both `text_1` and `text_2`, forming multiple sentence pairs
 where each pair is built from a string in `text_1` and the corresponding string in `text_2` (similar to `zip()`).
 The total number of pairs is `len(text_2)`.
 
-Request:
+??? Request
 
-```bash
+    ```bash
-curl -X 'POST' \
+    curl -X 'POST' \
       'http://127.0.0.1:8000/score' \
       -H 'accept: application/json' \
       -H 'Content-Type: application/json' \
@@ -611,13 +627,13 @@ curl -X 'POST' \
         "The capital of Brazil is Brasilia.",
         "The capital of France is Paris."
       ]
-}'
+    }'
-```
+    ```
 
-Response:
+??? Response
 
-```bash
+    ```bash
-{
+    {
       "id": "score-request-id",
       "object": "list",
       "created": 693447,
@@ -635,8 +651,8 @@ Response:
         }
       ],
       "usage": {}
-}
+    }
-```
+    ```
 
 #### Extra parameters
 
@@ -675,10 +691,10 @@ Code example: <gh-file:examples/online_serving/jinaai_rerank_client.py>
 Note that the `top_n` request parameter is optional and will default to the length of the `documents` field.
 Result documents will be sorted by relevance, and the `index` property can be used to determine original order.
 
-Request:
+??? Request
 
-```bash
+    ```bash
-curl -X 'POST' \
+    curl -X 'POST' \
       'http://127.0.0.1:8000/v1/rerank' \
       -H 'accept: application/json' \
       -H 'Content-Type: application/json' \
@@ -690,13 +706,13 @@ curl -X 'POST' \
         "The capital of France is Paris.",
         "Horses and cows are both animals"
       ]
-}'
+    }'
-```
+    ```
 
-Response:
+??? Response
 
-```bash
+    ```bash
-{
+    {
       "id": "rerank-fae51b2b664d4ed38f5969b612edff77",
       "model": "BAAI/bge-reranker-base",
       "usage": {
@@ -718,8 +734,8 @@ Response:
           "relevance_score": 0.0005860328674316406
         }
       ]
-}
+    }
-```
+    ```
 
 #### Extra parameters
 

docs/usage/metrics.md

@@ -12,28 +12,32 @@ vllm serve unsloth/Llama-3.2-1B-Instruct
 
 Then query the endpoint to get the latest metrics from the server:
 
-```console
+??? Output
-$ curl http://0.0.0.0:8000/metrics
 
-# HELP vllm:iteration_tokens_total Histogram of number of tokens per engine_step.
+    ```console
-# TYPE vllm:iteration_tokens_total histogram
+    $ curl http://0.0.0.0:8000/metrics
-vllm:iteration_tokens_total_sum{model_name="unsloth/Llama-3.2-1B-Instruct"} 0.0
+
-vllm:iteration_tokens_total_bucket{le="1.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
+    # HELP vllm:iteration_tokens_total Histogram of number of tokens per engine_step.
-vllm:iteration_tokens_total_bucket{le="8.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
+    # TYPE vllm:iteration_tokens_total histogram
-vllm:iteration_tokens_total_bucket{le="16.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
+    vllm:iteration_tokens_total_sum{model_name="unsloth/Llama-3.2-1B-Instruct"} 0.0
-vllm:iteration_tokens_total_bucket{le="32.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
+    vllm:iteration_tokens_total_bucket{le="1.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
-vllm:iteration_tokens_total_bucket{le="64.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
+    vllm:iteration_tokens_total_bucket{le="8.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
-vllm:iteration_tokens_total_bucket{le="128.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
+    vllm:iteration_tokens_total_bucket{le="16.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
-vllm:iteration_tokens_total_bucket{le="256.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
+    vllm:iteration_tokens_total_bucket{le="32.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
-vllm:iteration_tokens_total_bucket{le="512.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
+    vllm:iteration_tokens_total_bucket{le="64.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
-...
+    vllm:iteration_tokens_total_bucket{le="128.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
-```
+    vllm:iteration_tokens_total_bucket{le="256.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
+    vllm:iteration_tokens_total_bucket{le="512.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
+    ...
+    ```
 
 The following metrics are exposed:
 
-```python
+??? Code
---8<-- "vllm/engine/metrics.py:metrics-definitions"
+
-```
+    ```python
+    --8<-- "vllm/engine/metrics.py:metrics-definitions"
+    ```
 
 Note: when metrics are deprecated in version `X.Y`, they are hidden in version `X.Y+1`
 but can be re-enabled using the `--show-hidden-metrics-for-version=X.Y` escape hatch,

docs/usage/troubleshooting.md

@@ -60,68 +60,70 @@ To identify the particular CUDA operation that causes the error, you can add `--
 
 If GPU/CPU communication cannot be established, you can use the following Python script and follow the instructions below to confirm whether the GPU/CPU communication is working correctly.
 
-```python
+??? Code
-# Test PyTorch NCCL
-import torch
-import torch.distributed as dist
-dist.init_process_group(backend="nccl")
-local_rank = dist.get_rank() % torch.cuda.device_count()
-torch.cuda.set_device(local_rank)
-data = torch.FloatTensor([1,] * 128).to("cuda")
-dist.all_reduce(data, op=dist.ReduceOp.SUM)
-torch.cuda.synchronize()
-value = data.mean().item()
-world_size = dist.get_world_size()
-assert value == world_size, f"Expected {world_size}, got {value}"
 
-print("PyTorch NCCL is successful!")
+    ```python
+    # Test PyTorch NCCL
+    import torch
+    import torch.distributed as dist
+    dist.init_process_group(backend="nccl")
+    local_rank = dist.get_rank() % torch.cuda.device_count()
+    torch.cuda.set_device(local_rank)
+    data = torch.FloatTensor([1,] * 128).to("cuda")
+    dist.all_reduce(data, op=dist.ReduceOp.SUM)
+    torch.cuda.synchronize()
+    value = data.mean().item()
+    world_size = dist.get_world_size()
+    assert value == world_size, f"Expected {world_size}, got {value}"
 
-# Test PyTorch GLOO
+    print("PyTorch NCCL is successful!")
-gloo_group = dist.new_group(ranks=list(range(world_size)), backend="gloo")
-cpu_data = torch.FloatTensor([1,] * 128)
-dist.all_reduce(cpu_data, op=dist.ReduceOp.SUM, group=gloo_group)
-value = cpu_data.mean().item()
-assert value == world_size, f"Expected {world_size}, got {value}"
 
-print("PyTorch GLOO is successful!")
+    # Test PyTorch GLOO
+    gloo_group = dist.new_group(ranks=list(range(world_size)), backend="gloo")
+    cpu_data = torch.FloatTensor([1,] * 128)
+    dist.all_reduce(cpu_data, op=dist.ReduceOp.SUM, group=gloo_group)
+    value = cpu_data.mean().item()
+    assert value == world_size, f"Expected {world_size}, got {value}"
 
-if world_size <= 1:
+    print("PyTorch GLOO is successful!")
+
+    if world_size <= 1:
         exit()
 
-# Test vLLM NCCL, with cuda graph
+    # Test vLLM NCCL, with cuda graph
-from vllm.distributed.device_communicators.pynccl import PyNcclCommunicator
+    from vllm.distributed.device_communicators.pynccl import PyNcclCommunicator
 
-pynccl = PyNcclCommunicator(group=gloo_group, device=local_rank)
+    pynccl = PyNcclCommunicator(group=gloo_group, device=local_rank)
-# pynccl is enabled by default for 0.6.5+,
+    # pynccl is enabled by default for 0.6.5+,
-# but for 0.6.4 and below, we need to enable it manually.
+    # but for 0.6.4 and below, we need to enable it manually.
-# keep the code for backward compatibility when because people
+    # keep the code for backward compatibility when because people
-# prefer to read the latest documentation.
+    # prefer to read the latest documentation.
-pynccl.disabled = False
+    pynccl.disabled = False
 
-s = torch.cuda.Stream()
+    s = torch.cuda.Stream()
-with torch.cuda.stream(s):
+    with torch.cuda.stream(s):
         data.fill_(1)
         out = pynccl.all_reduce(data, stream=s)
         value = out.mean().item()
         assert value == world_size, f"Expected {world_size}, got {value}"
 
-print("vLLM NCCL is successful!")
+    print("vLLM NCCL is successful!")
 
-g = torch.cuda.CUDAGraph()
+    g = torch.cuda.CUDAGraph()
-with torch.cuda.graph(cuda_graph=g, stream=s):
+    with torch.cuda.graph(cuda_graph=g, stream=s):
         out = pynccl.all_reduce(data, stream=torch.cuda.current_stream())
 
-data.fill_(1)
+    data.fill_(1)
-g.replay()
+    g.replay()
-torch.cuda.current_stream().synchronize()
+    torch.cuda.current_stream().synchronize()
-value = out.mean().item()
+    value = out.mean().item()
-assert value == world_size, f"Expected {world_size}, got {value}"
+    assert value == world_size, f"Expected {world_size}, got {value}"
 
-print("vLLM NCCL with cuda graph is successful!")
+    print("vLLM NCCL with cuda graph is successful!")
 
-dist.destroy_process_group(gloo_group)
+    dist.destroy_process_group(gloo_group)
-dist.destroy_process_group()
+    dist.destroy_process_group()
-```
+    ```
 
 If you are testing with a single node, adjust `--nproc-per-node` to the number of GPUs you want to use:
 
@@ -165,8 +167,10 @@ WARNING 12-11 14:50:37 multiproc_worker_utils.py:281] CUDA was previously
 
 or an error from Python that looks like this:
 
-```console
+??? Logs
-RuntimeError:
+
+    ```console
+    RuntimeError:
             An attempt has been made to start a new process before the
             current process has finished its bootstrapping phase.
 
@@ -183,7 +187,7 @@ RuntimeError:
 
             To fix this issue, refer to the "Safe importing of main module"
             section in https://docs.python.org/3/library/multiprocessing.html
-```
+    ```
 
 then you must update your Python code to guard usage of `vllm` behind a `if
 __name__ == '__main__':` block. For example, instead of this:
@@ -207,20 +211,22 @@ if __name__ == '__main__':
 
 vLLM heavily depends on `torch.compile` to optimize the model for better performance, which introduces the dependency on the `torch.compile` functionality and the `triton` library. By default, we use `torch.compile` to [optimize some functions](https://github.com/vllm-project/vllm/pull/10406) in the model. Before running vLLM, you can check if `torch.compile` is working as expected by running the following script:
 
-```python
+??? Code
-import torch
 
-@torch.compile
+    ```python
-def f(x):
+    import torch
+
+    @torch.compile
+    def f(x):
         # a simple function to test torch.compile
         x = x + 1
         x = x * 2
         x = x.sin()
         return x
 
-x = torch.randn(4, 4).cuda()
+    x = torch.randn(4, 4).cuda()
-print(f(x))
+    print(f(x))
-```
+    ```
 
 If it raises errors from `torch/_inductor` directory, usually it means you have a custom `triton` library that is not compatible with the version of PyTorch you are using. See [this issue](https://github.com/vllm-project/vllm/issues/12219) for example.
 

docs/usage/usage_stats.md

@@ -10,8 +10,10 @@ The list of data collected by the latest version of vLLM can be found here: <gh-
 
 Here is an example as of v0.4.0:
 
-```json
+??? Output
-{
+
+    ```json
+    {
       "uuid": "fbe880e9-084d-4cab-a395-8984c50f1109",
       "provider": "GCP",
       "num_cpu": 24,
@@ -38,8 +40,8 @@ Here is an example as of v0.4.0:
       "enable_prefix_caching": false,
       "enforce_eager": false,
       "disable_custom_all_reduce": true
-}
+    }
-```
+    ```
 
 You can preview the collected data by running the following command: