[doc] Fold long code blocks to improve readability (#19926)

Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>

docs/ci/update_pytorch_version.md

@@ -91,7 +91,7 @@ source to unblock the update process.
 ### FlashInfer
 Here is how to build and install it from source with torch2.7.0+cu128 in vLLM [Dockerfile](https://github.com/vllm-project/vllm/blob/27bebcd89792d5c4b08af7a65095759526f2f9e1/docker/Dockerfile#L259-L271):
 
-```
+```bash
 export TORCH_CUDA_ARCH_LIST='7.5 8.0 8.9 9.0 10.0+PTX'
 export FLASHINFER_ENABLE_SM90=1
 uv pip install --system --no-build-isolation "git+https://github.com/flashinfer-ai/flashinfer@v0.2.6.post1"
@@ -105,14 +105,14 @@ team if you want to get the package published there.
 ### xFormers
 Similar to FlashInfer, here is how to build and install xFormers from source:
 
-```
+```bash
 export TORCH_CUDA_ARCH_LIST='7.0 7.5 8.0 8.9 9.0 10.0+PTX'
 MAX_JOBS=16 uv pip install --system --no-build-isolation "git+https://github.com/facebookresearch/xformers@v0.0.30"
 ```
 
 ### Mamba
 
-```
+```bash
 uv pip install --system --no-build-isolation "git+https://github.com/state-spaces/mamba@v2.2.4"
 ```
 

docs/cli/README.md

@@ -16,7 +16,7 @@ vllm {chat,complete,serve,bench,collect-env,run-batch}
 
 Start the vLLM OpenAI Compatible API server.
 
-Examples:
+??? Examples
 
     ```bash
     # Start with a model
@@ -43,8 +43,6 @@ vllm serve --help=max
 
 Generate chat completions via the running API server.
 
-Examples:
-
 ```bash
 # Directly connect to localhost API without arguments
 vllm chat
@@ -60,8 +58,6 @@ vllm chat --quick "hi"
 
 Generate text completions based on the given prompt via the running API server.
 
-Examples:
-
 ```bash
 # Directly connect to localhost API without arguments
 vllm complete
@@ -73,6 +69,8 @@ vllm complete --url http://{vllm-serve-host}:{vllm-serve-port}/v1
 vllm complete --quick "The future of AI is"
 ```
 
+</details>
+
 ## bench
 
 Run benchmark tests for latency online serving throughput and offline inference throughput.
@@ -89,8 +87,6 @@ vllm bench {latency, serve, throughput}
 
 Benchmark the latency of a single batch of requests.
 
-Example:
-
 ```bash
 vllm bench latency \
     --model meta-llama/Llama-3.2-1B-Instruct \
@@ -104,8 +100,6 @@ vllm bench latency \
 
 Benchmark the online serving throughput.
 
-Example:
-
 ```bash
 vllm bench serve \
     --model meta-llama/Llama-3.2-1B-Instruct \
@@ -120,8 +114,6 @@ vllm bench serve \
 
 Benchmark offline inference throughput.
 
-Example:
-
 ```bash
 vllm bench throughput \
     --model meta-llama/Llama-3.2-1B-Instruct \
@@ -143,7 +135,8 @@ vllm collect-env
 
 Run batch prompts and write results to file.
 
-Examples:
+<details>
+<summary>Examples</summary>
 
 ```bash
 # Running with a local file
@@ -159,6 +152,8 @@ vllm run-batch \
     --model meta-llama/Meta-Llama-3-8B-Instruct
 ```
 
+</details>
+
 ## More Help
 
 For detailed options of any subcommand, use:

docs/configuration/conserving_memory.md

@@ -57,6 +57,8 @@ By default, we optimize model inference using CUDA graphs which take up extra me
 
 You can adjust `compilation_config` to achieve a better balance between inference speed and memory usage:
 
+??? Code
+
     ```python
     from vllm import LLM
     from vllm.config import CompilationConfig, CompilationLevel
@@ -127,6 +129,8 @@ reduce the size of the processed multi-modal inputs, which in turn saves memory.
 
 Here are some examples:
 
+??? Code
+
     ```python
     from vllm import LLM
 

docs/configuration/env_vars.md

@@ -7,6 +7,8 @@ vLLM uses the following environment variables to configure the system:
 
     All environment variables used by vLLM are prefixed with `VLLM_`. **Special care should be taken for Kubernetes users**: please do not name the service as `vllm`, otherwise environment variables set by Kubernetes might conflict with vLLM's environment variables, because [Kubernetes sets environment variables for each service with the capitalized service name as the prefix](https://kubernetes.io/docs/concepts/services-networking/service/#environment-variables).
 
+??? Code
+
     ```python
     --8<-- "vllm/envs.py:env-vars-definition"
     ```

docs/contributing/README.md

@@ -93,6 +93,8 @@ For additional features and advanced configurations, refer to the official [MkDo
 
 ## Testing
 
+??? note "Commands"
+
     ```bash
     pip install -r requirements/dev.txt
 

docs/contributing/model/basic.md

@@ -27,6 +27,8 @@ All vLLM modules within the model must include a `prefix` argument in their cons
 
 The initialization code should look like this:
 
+??? Code
+
     ```python
     from torch import nn
     from vllm.config import VllmConfig

docs/contributing/model/multimodal.md

@@ -25,6 +25,8 @@ Further update the model as follows:
 
 - Implement [get_multimodal_embeddings][vllm.model_executor.models.interfaces.SupportsMultiModal.get_multimodal_embeddings] that returns the embeddings from running the multimodal inputs through the multimodal tokenizer of the model. Below we provide a boilerplate of a typical implementation pattern, but feel free to adjust it to your own needs.
 
+    ??? Code
+
         ```python
         class YourModelForImage2Seq(nn.Module):
             ...
@@ -53,6 +55,8 @@ Further update the model as follows:
 
 - Implement [get_input_embeddings][vllm.model_executor.models.interfaces.SupportsMultiModal.get_input_embeddings] to merge `multimodal_embeddings` with text embeddings from the `input_ids`. If input processing for the model is implemented correctly (see sections below), then you can leverage the utility function we provide to easily merge the embeddings.
 
+    ??? Code
+
         ```python
         from .utils import merge_multimodal_embeddings
 
@@ -135,6 +139,8 @@ Assuming that the memory usage increases with the number of tokens, the dummy in
 
     Looking at the code of HF's `LlavaForConditionalGeneration`:
 
+    ??? Code
+
         ```python
         # https://github.com/huggingface/transformers/blob/v4.47.1/src/transformers/models/llava/modeling_llava.py#L530-L544
         n_image_tokens = (input_ids == self.config.image_token_index).sum().item()
@@ -157,6 +163,8 @@ Assuming that the memory usage increases with the number of tokens, the dummy in
     The number of placeholder feature tokens per image is `image_features.shape[1]`.
     `image_features` is calculated inside the `get_image_features` method:
 
+    ??? Code
+
         ```python
         # https://github.com/huggingface/transformers/blob/v4.47.1/src/transformers/models/llava/modeling_llava.py#L290-L300
         image_outputs = self.vision_tower(pixel_values, output_hidden_states=True)
@@ -193,6 +201,8 @@ Assuming that the memory usage increases with the number of tokens, the dummy in
 
     To find the sequence length, we turn to the code of `CLIPVisionEmbeddings`:
 
+    ??? Code
+
         ```python
         # https://github.com/huggingface/transformers/blob/v4.47.1/src/transformers/models/clip/modeling_clip.py#L247-L257
         target_dtype = self.patch_embedding.weight.dtype
@@ -218,6 +228,8 @@ Assuming that the memory usage increases with the number of tokens, the dummy in
 
     Overall, the number of placeholder feature tokens for an image can be calculated as:
 
+    ??? Code
+
         ```python
         def get_num_image_tokens(
             self,
@@ -241,6 +253,8 @@ Assuming that the memory usage increases with the number of tokens, the dummy in
     Notice that the number of image tokens doesn't depend on the image width and height.
     We can simply use a dummy `image_size` to calculate the multimodal profiling data:
 
+    ??? Code
+
         ```python
         # NOTE: In actuality, this is usually implemented as part of the
         # model's subclass of `BaseProcessingInfo`, but we show it as is
@@ -284,6 +298,8 @@ Assuming that the memory usage increases with the number of tokens, the dummy in
 
     Looking at the code of HF's `FuyuForCausalLM`:
 
+    ??? Code
+
         ```python
         # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/modeling_fuyu.py#L311-L322
         if image_patches is not None and past_key_values is None:
@@ -312,6 +328,8 @@ Assuming that the memory usage increases with the number of tokens, the dummy in
     In `FuyuImageProcessor.preprocess`, the images are resized and padded to the target `FuyuImageProcessor.size`,
     returning the dimensions after resizing (but before padding) as metadata.
 
+    ??? Code
+
         ```python
         # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/processing_fuyu.py#L541-L544
         image_encoding = self.image_processor.preprocess(images, **output_kwargs["images_kwargs"])
@@ -348,6 +366,8 @@ Assuming that the memory usage increases with the number of tokens, the dummy in
 
     In `FuyuImageProcessor.preprocess_with_tokenizer_info`, the images are split into patches based on this metadata:
 
+    ??? Code
+
         ```python
         # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/processing_fuyu.py#L417-L425
         model_image_input = self.image_processor.preprocess_with_tokenizer_info(
@@ -384,6 +404,8 @@ Assuming that the memory usage increases with the number of tokens, the dummy in
 
     The number of patches is in turn defined by `FuyuImageProcessor.get_num_patches`:
 
+    ??? Code
+
         ```python
         # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/image_processing_fuyu.py#L552-L562
         patch_size = patch_size if patch_size is not None else self.patch_size
@@ -419,6 +441,8 @@ Assuming that the memory usage increases with the number of tokens, the dummy in
 
     For the multimodal image profiling data, the logic is very similar to LLaVA:
 
+    ??? Code
+
         ```python
         def get_dummy_mm_data(
             self,
@@ -455,6 +479,7 @@ return a schema of the tensors outputted by the HF processor that are related to
     The output of `CLIPImageProcessor` is a simple tensor with shape
     `(num_images, num_channels, image_height, image_width)`:
 
+
     ```python
     # https://github.com/huggingface/transformers/blob/v4.47.1/src/transformers/models/clip/image_processing_clip.py#L339-L345
     images = [
@@ -505,6 +530,8 @@ return a schema of the tensors outputted by the HF processor that are related to
     In order to support the use of [MultiModalFieldConfig.batched][] like in LLaVA,
     we remove the extra batch dimension by overriding [BaseMultiModalProcessor._call_hf_processor][]:
 
+    ??? Code
+
         ```python
         def _call_hf_processor(
             self,
@@ -573,6 +600,8 @@ Each [PromptUpdate][vllm.multimodal.processing.PromptUpdate] instance specifies
     It simply repeats each input `image_token` a number of times equal to the number of placeholder feature tokens (`num_image_tokens`).
     Based on this, we override [_get_prompt_updates][vllm.multimodal.processing.BaseMultiModalProcessor._get_prompt_updates] as follows:
 
+    ??? Code
+
         ```python
         def _get_prompt_updates(
             self,
@@ -616,6 +645,8 @@ Each [PromptUpdate][vllm.multimodal.processing.PromptUpdate] instance specifies
 
     We define a helper function to return `ncols` and `nrows` directly:
 
+    ??? Code
+
         ```python
         def get_image_feature_grid_size(
             self,
@@ -644,6 +675,8 @@ Each [PromptUpdate][vllm.multimodal.processing.PromptUpdate] instance specifies
 
     Based on this, we can initially define our replacement tokens as:
 
+    ??? Code
+
         ```python
         def get_replacement(item_idx: int):
             images = mm_items.get_items("image", ImageProcessorItems)
@@ -662,6 +695,8 @@ Each [PromptUpdate][vllm.multimodal.processing.PromptUpdate] instance specifies
     However, this is not entirely correct. After `FuyuImageProcessor.preprocess_with_tokenizer_info` is called,
     a BOS token (`<s>`) is also added to the promopt:
 
+    ??? Code
+
         ```python
         # https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/processing_fuyu.py#L417-L435
         model_image_input = self.image_processor.preprocess_with_tokenizer_info(
@@ -687,6 +722,8 @@ Each [PromptUpdate][vllm.multimodal.processing.PromptUpdate] instance specifies
     To assign the vision embeddings to only the image tokens, instead of a string
     you can return an instance of [PromptUpdateDetails][vllm.multimodal.processing.PromptUpdateDetails]:
 
+    ??? Code
+
         ```python
         hf_config = self.info.get_hf_config()
         bos_token_id = hf_config.bos_token_id  # `<s>`
@@ -712,6 +749,8 @@ Each [PromptUpdate][vllm.multimodal.processing.PromptUpdate] instance specifies
     Finally, noticing that the HF processor removes the `|ENDOFTEXT|` token from the tokenized prompt,
     we can search for it to conduct the replacement at the start of the string:
 
+    ??? Code
+
         ```python
         def _get_prompt_updates(
             self,

docs/contributing/profiling.md

@@ -97,7 +97,7 @@ to manually kill the profiler and generate your `nsys-rep` report.
 
 You can view these profiles either as summaries in the CLI, using `nsys stats [profile-file]`, or in the GUI by installing Nsight [locally following the directions here](https://developer.nvidia.com/nsight-systems/get-started).
 
-CLI example:
+??? CLI example
 
     ```bash
     nsys stats report1.nsys-rep

docs/deployment/docker.md

@@ -97,6 +97,8 @@ of PyTorch Nightly and should be considered **experimental**. Using the flag `--
     flags to speed up build process. However, ensure your `max_jobs` is substantially larger than `nvcc_threads` to get the most benefits.
     Keep an eye on memory usage with parallel jobs as it can be substantial (see example below).
 
+??? Command
+
     ```console
     # Example of building on Nvidia GH200 server. (Memory usage: ~15GB, Build time: ~1475s / ~25 min, Image size: 6.93GB)
     python3 use_existing_torch.py

docs/deployment/frameworks/autogen.md

@@ -30,6 +30,8 @@ python -m vllm.entrypoints.openai.api_server \
 
 - Call it with AutoGen:
 
+??? Code
+
     ```python
     import asyncio
     from autogen_core.models import UserMessage

docs/deployment/frameworks/cerebrium.md

@@ -34,6 +34,8 @@ vllm = "latest"
 
 Next, let us add our code to handle inference for the LLM of your choice (`mistralai/Mistral-7B-Instruct-v0.1` for this example), add the following code to your `main.py`:
 
+??? Code
+
     ```python
     from vllm import LLM, SamplingParams
 
@@ -62,6 +64,8 @@ cerebrium deploy
 
 If successful, you should be returned a CURL command that you can call inference against. Just remember to end the url with the function name you are calling (in our case`/run`)
 
+??? Command
+
     ```python
     curl -X POST https://api.cortex.cerebrium.ai/v4/p-xxxxxx/vllm/run \
     -H 'Content-Type: application/json' \
@@ -78,6 +82,8 @@ curl -X POST https://api.cortex.cerebrium.ai/v4/p-xxxxxx/vllm/run \
 
 You should get a response like:
 
+??? Response
+
     ```python
     {
         "run_id": "52911756-3066-9ae8-bcc9-d9129d1bd262",

docs/deployment/frameworks/dstack.md

@@ -26,6 +26,8 @@ dstack init
 
 Next, to provision a VM instance with LLM of your choice (`NousResearch/Llama-2-7b-chat-hf` for this example), create the following `serve.dstack.yml` file for the dstack `Service`:
 
+??? Config
+
     ```yaml
     type: service
 
@@ -46,6 +48,8 @@ model:
 
 Then, run the following CLI for provisioning:
 
+??? Command
+
     ```console
     $ dstack run . -f serve.dstack.yml
 
@@ -75,6 +79,8 @@ Service is published at ...
 
 After the provisioning, you can interact with the model by using the OpenAI SDK:
 
+??? Code
+
     ```python
     from openai import OpenAI
 

docs/deployment/frameworks/haystack.md

@@ -27,6 +27,8 @@ vllm serve mistralai/Mistral-7B-Instruct-v0.1
 
 - Use the `OpenAIGenerator` and `OpenAIChatGenerator` components in Haystack to query the vLLM server.
 
+??? Code
+
     ```python
     from haystack.components.generators.chat import OpenAIChatGenerator
     from haystack.dataclasses import ChatMessage
@@ -49,8 +51,6 @@ print(response)
     print("-"*30)
     ```
 
-Output e.g.:
-
 ```console
 ------------------------------
 {'replies': [ChatMessage(_role=<ChatRole.ASSISTANT: 'assistant'>, _content=[TextContent(text=' Of course! Where in Italy would you like to go and what type of trip are you looking to plan?')], _name=None, _meta={'model': 'mistralai/Mistral-7B-Instruct-v0.1', 'index': 0, 'finish_reason': 'stop', 'usage': {'completion_tokens': 23, 'prompt_tokens': 21, 'total_tokens': 44, 'completion_tokens_details': None, 'prompt_tokens_details': None}})]}

docs/deployment/frameworks/litellm.md

@@ -34,6 +34,8 @@ vllm serve qwen/Qwen1.5-0.5B-Chat
 
 - Call it with litellm:
 
+??? Code
+
     ```python
     import litellm 
 

docs/deployment/frameworks/lws.md

@@ -17,6 +17,8 @@ vLLM can be deployed with [LWS](https://github.com/kubernetes-sigs/lws) on Kuber
 
 Deploy the following yaml file `lws.yaml`
 
+??? Yaml
+
     ```yaml
     apiVersion: leaderworkerset.x-k8s.io/v1
     kind: LeaderWorkerSet
@@ -175,6 +177,8 @@ curl http://localhost:8080/v1/completions \
 
 The output should be similar to the following
 
+??? Output
+
     ```text
     {
       "id": "cmpl-1bb34faba88b43f9862cfbfb2200949d",

docs/deployment/frameworks/skypilot.md

@@ -24,6 +24,8 @@ sky check
 
 See the vLLM SkyPilot YAML for serving, [serving.yaml](https://github.com/skypilot-org/skypilot/blob/master/llm/vllm/serve.yaml).
 
+??? Yaml
+
     ```yaml
     resources:
       accelerators: {L4, A10g, A10, L40, A40, A100, A100-80GB} # We can use cheaper accelerators for 8B model.
@@ -93,6 +95,8 @@ HF_TOKEN="your-huggingface-token" \
 
 SkyPilot can scale up the service to multiple service replicas with built-in autoscaling, load-balancing and fault-tolerance. You can do it by adding a services section to the YAML file.
 
+??? Yaml
+
     ```yaml
     service:
       replicas: 2
@@ -107,8 +111,7 @@ service:
       max_completion_tokens: 1
     ```
 
-<details>
+??? Yaml
-<summary>Click to see the full recipe YAML</summary>
 
     ```yaml
     service:
@@ -154,8 +157,6 @@ run: |
         2>&1 | tee api_server.log
     ```
 
-</details>
-
 Start the serving the Llama-3 8B model on multiple replicas:
 
 ```console
@@ -170,8 +171,7 @@ Wait until the service is ready:
 watch -n10 sky serve status vllm
 ```
 
-<details>
+Example outputs:
-<summary>Example outputs:</summary>
 
 ```console
 Services
@@ -184,11 +184,11 @@ vllm          1   1        xx.yy.zz.121  18 mins ago  1x GCP([Spot]{'L4': 1})  R
 vllm          2   1        xx.yy.zz.245  18 mins ago  1x GCP([Spot]{'L4': 1})  READY   us-east4
 ```
 
-</details>
-
 After the service is READY, you can find a single endpoint for the service and access the service with the endpoint:
 
-```console
+??? Commands
+
+    ```bash
     ENDPOINT=$(sky serve status --endpoint 8081 vllm)
     curl -L http://$ENDPOINT/v1/chat/completions \
       -H "Content-Type: application/json" \
@@ -220,8 +220,7 @@ service:
 
 This will scale the service up to when the QPS exceeds 2 for each replica.
 
-<details>
+??? Yaml
-<summary>Click to see the full recipe YAML</summary>
 
     ```yaml
     service:
@@ -270,8 +269,6 @@ run: |
         2>&1 | tee api_server.log
     ```
 
-</details>
-
 To update the service with the new config:
 
 ```console
@@ -288,8 +285,7 @@ sky serve down vllm
 
 It is also possible to access the Llama-3 service with a separate GUI frontend, so the user requests send to the GUI will be load-balanced across replicas.
 
-<details>
+??? Yaml
-<summary>Click to see the full GUI YAML</summary>
 
     ```yaml
     envs:
@@ -319,8 +315,6 @@ run: |
         --stop-token-ids 128009,128001 | tee ~/gradio.log
     ```
 
-</details>
-
 1. Start the chat web UI:
 
     ```console

docs/deployment/integrations/production-stack.md

@@ -60,7 +60,7 @@ And then you can send out a query to the OpenAI-compatible API to check the avai
 curl -o- http://localhost:30080/models
 ```
 
-Expected output:
+??? Output
 
     ```json
     {
@@ -89,7 +89,7 @@ curl -X POST http://localhost:30080/completions \
   }'
 ```
 
-Expected output:
+??? Output
 
     ```json
     {
@@ -121,6 +121,8 @@ sudo helm uninstall vllm
 
 The core vLLM production stack configuration is managed with YAML. Here is the example configuration used in the installation above:
 
+??? Yaml
+
     ```yaml
     servingEngineSpec:
       runtimeClassName: ""

docs/deployment/k8s.md

@@ -29,6 +29,8 @@ Alternatively, you can deploy vLLM to Kubernetes using any of the following:
 
 First, create a Kubernetes PVC and Secret for downloading and storing Hugging Face model:
 
+??? Config
+
     ```bash
     cat <<EOF |kubectl apply -f -
     apiVersion: v1
@@ -55,6 +57,8 @@ EOF
 
 Next, start the vLLM server as a Kubernetes Deployment and Service:
 
+??? Config
+
     ```bash
     cat <<EOF |kubectl apply -f -
     apiVersion: apps/v1
@@ -128,6 +132,9 @@ INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
 
       PVC is used to store the model cache and it is optional, you can use hostPath or other storage options
 
+      <details>
+      <summary>Yaml</summary>
+
       ```yaml
       apiVersion: v1
       kind: PersistentVolumeClaim
@@ -144,6 +151,8 @@ INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
         volumeMode: Filesystem
       ```
 
+      </details>
+
       Secret is optional and only required for accessing gated models, you can skip this step if you are not using gated models
 
       ```yaml
@@ -163,6 +172,9 @@ INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
 
       NVIDIA GPU:
 
+      <details>
+      <summary>Yaml</summary>
+
       ```yaml
       apiVersion: apps/v1
       kind: Deployment
@@ -233,10 +245,15 @@ INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
                 periodSeconds: 5
       ```
 
+      </details>
+
       AMD GPU:
 
       You can refer to the `deployment.yaml` below if using AMD ROCm GPU like MI300X.
 
+      <details>
+      <summary>Yaml</summary>
+
       ```yaml
       apiVersion: apps/v1
       kind: Deployment
@@ -305,12 +322,17 @@ INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
                 mountPath: /dev/shm
       ```
 
+      </details>
+
       You can get the full example with steps and sample yaml files from <https://github.com/ROCm/k8s-device-plugin/tree/master/example/vllm-serve>.
 
 2. Create a Kubernetes Service for vLLM
 
       Next, create a Kubernetes Service file to expose the `mistral-7b` deployment:
 
+      <details>
+      <summary>Yaml</summary>
+
       ```yaml
       apiVersion: v1
       kind: Service
@@ -330,6 +352,8 @@ INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
         type: ClusterIP
       ```
 
+      </details>
+
 3. Deploy and Test
 
       Apply the deployment and service configurations using `kubectl apply -f <filename>`:

docs/deployment/nginx.md

@@ -36,6 +36,8 @@ docker build . -f Dockerfile.nginx --tag nginx-lb
 
 Create a file named `nginx_conf/nginx.conf`. Note that you can add as many servers as you'd like. In the below example we'll start with two. To add more, add another `server vllmN:8000 max_fails=3 fail_timeout=10000s;` entry to `upstream backend`.
 
+??? Config
+
     ```console
     upstream backend {
         least_conn;
@@ -93,6 +95,8 @@ Notes:
 - The below example assumes GPU backend used. If you are using CPU backend, remove `--gpus device=ID`, add `VLLM_CPU_KVCACHE_SPACE` and `VLLM_CPU_OMP_THREADS_BIND` environment variables to the docker run command.
 - Adjust the model name that you want to use in your vLLM servers if you don't want to use `Llama-2-7b-chat-hf`.
 
+??? Commands
+
     ```console
     mkdir -p ~/.cache/huggingface/hub/
     hf_cache_dir=~/.cache/huggingface/

docs/design/arch_overview.md

@@ -22,6 +22,8 @@ server.
 
 Here is a sample of `LLM` class usage:
 
+??? Code
+
     ```python
     from vllm import LLM, SamplingParams
 
@@ -178,6 +180,8 @@ vision-language model.
 
     To avoid accidentally passing incorrect arguments, the constructor is now keyword-only. This ensures that the constructor will raise an error if old configurations are passed. vLLM developers have already made this change for all models within vLLM. For out-of-tree registered models, developers need to update their models, for example by adding shim code to adapt the old constructor signature to the new one:
 
+    ??? Code
+
         ```python
         class MyOldModel(nn.Module):
             def __init__(

docs/design/kernel/paged_attention.md

@@ -448,6 +448,8 @@ elements of the entire head for all context tokens. However, overall,
 all results for output have been calculated but are just stored in
 different thread register memory.
 
+??? Code
+
     ```cpp
     float* out_smem = reinterpret_cast<float*>(shared_mem);
     for (int i = NUM_WARPS; i > 1; i /= 2) {

docs/design/plugin_system.md

@@ -13,6 +13,8 @@ Plugins are user-registered code that vLLM executes. Given vLLM's architecture (
 
 vLLM's plugin system uses the standard Python `entry_points` mechanism. This mechanism allows developers to register functions in their Python packages for use by other packages. An example of a plugin:
 
+??? Code
+
     ```python
     # inside `setup.py` file
     from setuptools import setup

docs/features/lora.md

@@ -29,6 +29,8 @@ We can now submit the prompts and call `llm.generate` with the `lora_request` pa
 of `LoRARequest` is a human identifiable name, the second parameter is a globally unique ID for the adapter and
 the third parameter is the path to the LoRA adapter.
 
+??? Code
+
     ```python
     sampling_params = SamplingParams(
         temperature=0,
@@ -68,6 +70,8 @@ The server entrypoint accepts all other LoRA configuration parameters (`max_lora
 etc.), which will apply to all forthcoming requests. Upon querying the `/models` endpoint, we should see our LoRA along
 with its base model (if `jq` is not installed, you can follow [this guide](https://jqlang.org/download/) to install it.):
 
+??? Command
+
     ```bash
     curl localhost:8000/v1/models | jq .
     {
@@ -168,7 +172,7 @@ Alternatively, follow these example steps to implement your own plugin:
 
 1. Implement the LoRAResolver interface.
 
-    Example of a simple S3 LoRAResolver implementation:
+    ??? Example of a simple S3 LoRAResolver implementation
 
         ```python
         import os
@@ -234,6 +238,8 @@ The new format of `--lora-modules` is mainly to support the display of parent mo
 - The `parent` field of LoRA model `sql-lora` now links to its base model `meta-llama/Llama-2-7b-hf`. This correctly reflects the hierarchical relationship between the base model and the LoRA adapter.
 - The `root` field points to the artifact location of the lora adapter.
 
+??? Command output
+
     ```bash
     $ curl http://localhost:8000/v1/models
 

docs/features/multimodal_inputs.md

@@ -20,6 +20,8 @@ To input multi-modal data, follow this schema in [vllm.inputs.PromptType][]:
 
 You can pass a single image to the `'image'` field of the multi-modal dictionary, as shown in the following examples:
 
+??? Code
+
     ```python
     from vllm import LLM
 
@@ -66,6 +68,8 @@ Full example: <gh-file:examples/offline_inference/vision_language.py>
 
 To substitute multiple images inside the same text prompt, you can pass in a list of images instead:
 
+??? Code
+
     ```python
     from vllm import LLM
 
@@ -99,6 +103,8 @@ Full example: <gh-file:examples/offline_inference/vision_language_multi_image.py
 
 Multi-image input can be extended to perform video captioning. We show this with [Qwen2-VL](https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct) as it supports videos:
 
+??? Code
+
     ```python
     from vllm import LLM
 
@@ -144,6 +150,8 @@ Full example: <gh-file:examples/offline_inference/audio_language.py>
 To input pre-computed embeddings belonging to a data type (i.e. image, video, or audio) directly to the language model,
 pass a tensor of shape `(num_items, feature_size, hidden_size of LM)` to the corresponding field of the multi-modal dictionary.
 
+??? Code
+
     ```python
     from vllm import LLM
 
@@ -169,6 +177,8 @@ for o in outputs:
 
 For Qwen2-VL and MiniCPM-V, we accept additional parameters alongside the embeddings:
 
+??? Code
+
     ```python
     # Construct the prompt based on your model
     prompt = ...
@@ -235,6 +245,8 @@ vllm serve microsoft/Phi-3.5-vision-instruct --task generate \
 
 Then, you can use the OpenAI client as follows:
 
+??? Code
+
     ```python
     from openai import OpenAI
 
@@ -311,6 +323,8 @@ vllm serve llava-hf/llava-onevision-qwen2-0.5b-ov-hf --task generate --max-model
 
 Then, you can use the OpenAI client as follows:
 
+??? Code
+
     ```python
     from openai import OpenAI
 
@@ -373,6 +387,8 @@ vllm serve fixie-ai/ultravox-v0_5-llama-3_2-1b
 
 Then, you can use the OpenAI client as follows:
 
+??? Code
+
     ```python
     import base64
     import requests
@@ -427,6 +443,8 @@ print("Chat completion output from input audio:", result)
 
 Alternatively, you can pass `audio_url`, which is the audio counterpart of `image_url` for image input:
 
+??? Code
+
     ```python
     chat_completion_from_url = client.chat.completions.create(
         messages=[{
@@ -470,6 +488,8 @@ pass a tensor of shape to the corresponding field of the multi-modal dictionary.
 For image embeddings, you can pass the base64-encoded tensor to the `image_embeds` field.
 The following example demonstrates how to pass image embeddings to the OpenAI server:
 
+??? Code
+
     ```python
     image_embedding = torch.load(...)
     grid_thw = torch.load(...) # Required by Qwen/Qwen2-VL-2B-Instruct

docs/features/quantization/auto_awq.md

@@ -15,6 +15,8 @@ pip install autoawq
 
 After installing AutoAWQ, you are ready to quantize a model. Please refer to the [AutoAWQ documentation](https://casper-hansen.github.io/AutoAWQ/examples/#basic-quantization) for further details. Here is an example of how to quantize `mistralai/Mistral-7B-Instruct-v0.2`:
 
+??? Code
+
     ```python
     from awq import AutoAWQForCausalLM
     from transformers import AutoTokenizer
@@ -49,6 +51,8 @@ python examples/offline_inference/llm_engine_example.py \
 
 AWQ models are also supported directly through the LLM entrypoint:
 
+??? Code
+
     ```python
     from vllm import LLM, SamplingParams
 

docs/features/quantization/bitblas.md

@@ -43,6 +43,8 @@ llm = LLM(
 
 ## Read gptq format checkpoint
 
+??? Code
+
     ```python
     from vllm import LLM
     import torch

docs/features/quantization/fp8.md

@@ -58,6 +58,8 @@ For FP8 quantization, we can recover accuracy with simple RTN quantization. We r
 
 Since simple RTN does not require data for weight quantization and the activations are quantized dynamically, we do not need any calibration data for this quantization flow.
 
+??? Code
+
     ```python
     from llmcompressor.transformers import oneshot
     from llmcompressor.modifiers.quantization import QuantizationModifier

docs/features/quantization/gguf.md

@@ -41,6 +41,8 @@ vllm serve ./tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf \
 
 You can also use the GGUF model directly through the LLM entrypoint:
 
+??? Code
+
       ```python
       from vllm import LLM, SamplingParams
 

docs/features/quantization/gptqmodel.md

@@ -31,6 +31,8 @@ After installing GPTQModel, you are ready to quantize a model. Please refer to t
 
 Here is an example of how to quantize `meta-llama/Llama-3.2-1B-Instruct`:
 
+??? Code
+
     ```python
     from datasets import load_dataset
     from gptqmodel import GPTQModel, QuantizeConfig
@@ -67,6 +69,8 @@ python examples/offline_inference/llm_engine_example.py \
 
 GPTQModel quantized models are also supported directly through the LLM entrypoint:
 
+??? Code
+
     ```python
     from vllm import LLM, SamplingParams
 

docs/features/quantization/int4.md

@@ -53,6 +53,8 @@ When quantizing weights to INT4, you need sample data to estimate the weight upd
 It's best to use calibration data that closely matches your deployment data.
 For a general-purpose instruction-tuned model, you can use a dataset like `ultrachat`:
 
+??? Code
+
     ```python
     from datasets import load_dataset
 
@@ -76,6 +78,8 @@ ds = ds.map(tokenize, remove_columns=ds.column_names)
 
 Now, apply the quantization algorithms:
 
+??? Code
+
     ```python
     from llmcompressor.transformers import oneshot
     from llmcompressor.modifiers.quantization import GPTQModifier
@@ -137,6 +141,8 @@ $ lm_eval --model vllm \
 
 The following is an example of an expanded quantization recipe you can tune to your own use case:
 
+??? Code
+
     ```python
     from compressed_tensors.quantization import (
         QuantizationArgs,

docs/features/quantization/int8.md

@@ -54,6 +54,8 @@ When quantizing activations to INT8, you need sample data to estimate the activa
 It's best to use calibration data that closely matches your deployment data.
 For a general-purpose instruction-tuned model, you can use a dataset like `ultrachat`:
 
+??? Code
+
     ```python
     from datasets import load_dataset
 
@@ -73,10 +75,14 @@ def tokenize(sample):
     ds = ds.map(tokenize, remove_columns=ds.column_names)
     ```
 
+</details>
+
 ### 3. Applying Quantization
 
 Now, apply the quantization algorithms:
 
+??? Code
+
     ```python
     from llmcompressor.transformers import oneshot
     from llmcompressor.modifiers.quantization import GPTQModifier

docs/features/quantization/modelopt.md

@@ -14,6 +14,8 @@ You can quantize HuggingFace models using the example scripts provided in the Te
 
 Below is an example showing how to quantize a model using modelopt's PTQ API:
 
+??? Code
+
     ```python
     import modelopt.torch.quantization as mtq
     from transformers import AutoModelForCausalLM
@@ -48,6 +50,8 @@ with torch.inference_mode():
 
 The quantized checkpoint can then be deployed with vLLM. As an example, the following code shows how to deploy `nvidia/Llama-3.1-8B-Instruct-FP8`, which is the FP8 quantized checkpoint derived from `meta-llama/Llama-3.1-8B-Instruct`, using vLLM:
 
+??? Code
+
     ```python
     from vllm import LLM, SamplingParams
 

docs/features/quantization/quantized_kvcache.md

@@ -35,6 +35,8 @@ Studies have shown that FP8 E4M3 quantization typically only minimally degrades
 
 Here is an example of how to enable FP8 quantization:
 
+??? Code
+
     ```python
     # To calculate kv cache scales on the fly enable the calculate_kv_scales
     # parameter
@@ -71,6 +73,8 @@ pip install llmcompressor
 
 Here's a complete example using `meta-llama/Llama-3.1-8B-Instruct` (most models can use this same pattern):
 
+??? Code
+
     ```python
     from datasets import load_dataset
     from transformers import AutoModelForCausalLM, AutoTokenizer

docs/features/quantization/quark.md

@@ -42,6 +42,8 @@ The Quark quantization process can be listed for 5 steps as below:
 Quark uses [Transformers](https://huggingface.co/docs/transformers/en/index)
 to fetch model and tokenizer.
 
+??? Code
+
     ```python
     from transformers import AutoTokenizer, AutoModelForCausalLM
 
@@ -63,6 +65,8 @@ Quark uses the [PyTorch Dataloader](https://pytorch.org/tutorials/beginner/basic
 to load calibration data. For more details about how to use calibration datasets efficiently, please refer
 to [Adding Calibration Datasets](https://quark.docs.amd.com/latest/pytorch/calibration_datasets.html).
 
+??? Code
+
     ```python
     from datasets import load_dataset
     from torch.utils.data import DataLoader
@@ -94,6 +98,8 @@ kv-cache and the quantization algorithm is AutoSmoothQuant.
     AutoSmoothQuant config file for Llama is
     `examples/torch/language_modeling/llm_ptq/models/llama/autosmoothquant_config.json`.
 
+??? Code
+
     ```python
     from quark.torch.quantization import (Config, QuantizationConfig,
                                         FP8E4M3PerTensorSpec,
@@ -139,6 +145,8 @@ HuggingFace `safetensors`, you can refer to
 [HuggingFace format exporting](https://quark.docs.amd.com/latest/pytorch/export/quark_export_hf.html)
 for more exporting format details.
 
+??? Code
+
     ```python
     import torch
     from quark.torch import ModelQuantizer, ModelExporter
@@ -168,6 +176,8 @@ with torch.no_grad():
 
 Now, you can load and run the Quark quantized model directly through the LLM entrypoint:
 
+??? Code
+
     ```python
     from vllm import LLM, SamplingParams
 

docs/features/quantization/torchao.md

@@ -15,6 +15,8 @@ pip install \
 ## Quantizing HuggingFace Models
 You can quantize your own huggingface model with torchao, e.g. [transformers](https://huggingface.co/docs/transformers/main/en/quantization/torchao) and [diffusers](https://huggingface.co/docs/diffusers/en/quantization/torchao), and save the checkpoint to huggingface hub like [this](https://huggingface.co/jerryzh168/llama3-8b-int8wo) with the following example code:
 
+??? Code
+
     ```Python
     import torch
     from transformers import TorchAoConfig, AutoModelForCausalLM, AutoTokenizer

docs/features/reasoning_outputs.md

@@ -33,6 +33,8 @@ vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \
 
 Next, make a request to the model that should return the reasoning content in the response.
 
+??? Code
+
     ```python
     from openai import OpenAI
 
@@ -68,6 +70,8 @@ The `reasoning_content` field contains the reasoning steps that led to the final
 
 Streaming chat completions are also supported for reasoning models. The `reasoning_content` field is available in the `delta` field in [chat completion response chunks](https://platform.openai.com/docs/api-reference/chat/streaming).
 
+??? Json
+
     ```json
     {
         "id": "chatcmpl-123",
@@ -91,6 +95,8 @@ Streaming chat completions are also supported for reasoning models. The `reasoni
 
 OpenAI Python client library does not officially support `reasoning_content` attribute for streaming output. But the client supports extra attributes in the response. You can use `hasattr` to check if the `reasoning_content` attribute is present in the response. For example:
 
+??? Code
+
     ```python
     from openai import OpenAI
 
@@ -146,6 +152,8 @@ Remember to check whether the `reasoning_content` exists in the response before
 
 The reasoning content is also available when both tool calling and the reasoning parser are enabled. Additionally, tool calling only parses functions from the `content` field, not from the `reasoning_content`.
 
+??? Code
+
     ```python
     from openai import OpenAI
 
@@ -192,6 +200,8 @@ For more examples, please refer to <gh-file:examples/online_serving/openai_chat_
 
 You can add a new `ReasoningParser` similar to <gh-file:vllm/reasoning/deepseek_r1_reasoning_parser.py>.
 
+??? Code
+
     ```python
     # import the required packages
 
@@ -248,6 +258,8 @@ class ExampleParser(ReasoningParser):
 
 Additionally, to enable structured output, you'll need to create a new `Reasoner` similar to the one in <gh-file:vllm/reasoning/deepseek_r1_reasoning_parser.py>.
 
+??? Code
+
     ```python
     @dataclass
     class DeepSeekReasoner(Reasoner):

docs/features/spec_decode.md

@@ -18,6 +18,8 @@ Speculative decoding is a technique which improves inter-token latency in memory
 
 The following code configures vLLM in an offline mode to use speculative decoding with a draft model, speculating 5 tokens at a time.
 
+??? Code
+
     ```python
     from vllm import LLM, SamplingParams
 
@@ -60,6 +62,8 @@ python -m vllm.entrypoints.openai.api_server \
 
 Then use a client:
 
+??? Code
+
     ```python
     from openai import OpenAI
 
@@ -99,6 +103,8 @@ else:
 The following code configures vLLM to use speculative decoding where proposals are generated by
 matching n-grams in the prompt. For more information read [this thread.](https://x.com/joao_gante/status/1747322413006643259)
 
+??? Code
+
     ```python
     from vllm import LLM, SamplingParams
 
@@ -131,6 +137,8 @@ draft models that conditioning draft predictions on both context vectors and sam
 For more information see [this blog](https://pytorch.org/blog/hitchhikers-guide-speculative-decoding/) or
 [this technical report](https://arxiv.org/abs/2404.19124).
 
+??? Code
+
     ```python
     from vllm import LLM, SamplingParams
 
@@ -177,6 +185,8 @@ A variety of speculative models of this type are available on HF hub:
 The following code configures vLLM to use speculative decoding where proposals are generated by
 an [EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency)](https://arxiv.org/pdf/2401.15077) based draft model. A more detailed example for offline mode, including how to extract request level acceptance rate, can be found [here](gh-file:examples/offline_inference/eagle.py).
 
+??? Code
+
     ```python
     from vllm import LLM, SamplingParams
 

docs/features/structured_outputs.md

@@ -33,6 +33,8 @@ text.
 
 Now let´s see an example for each of the cases, starting with the `guided_choice`, as it´s the easiest one:
 
+??? Code
+
     ```python
     from openai import OpenAI
     client = OpenAI(
@@ -53,6 +55,8 @@ print(completion.choices[0].message.content)
 
 The next example shows how to use the `guided_regex`. The idea is to generate an email address, given a simple regex template:
 
+??? Code
+
     ```python
     completion = client.chat.completions.create(
         model=model,
@@ -75,6 +79,8 @@ For this we can use the `guided_json` parameter in two different ways:
 
 The next example shows how to use the `guided_json` parameter with a Pydantic model:
 
+??? Code
+
     ```python
     from pydantic import BaseModel
     from enum import Enum
@@ -121,6 +127,8 @@ difficult to use, but it´s really powerful. It allows us to define complete
 languages like SQL queries. It works by using a context free EBNF grammar.
 As an example, we can use to define a specific format of simplified SQL queries:
 
+??? Code
+
     ```python
     simplified_sql_grammar = """
         root ::= select_statement
@@ -161,6 +169,8 @@ vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --reasoning-parser deepseek_r
 
 Note that you can use reasoning with any provided structured outputs feature. The following uses one with JSON schema:
 
+??? Code
+
     ```python
     from pydantic import BaseModel
 
@@ -202,6 +212,8 @@ For the following examples, vLLM was setup using `vllm serve meta-llama/Llama-3.
 
 Here is a simple example demonstrating how to get structured output using Pydantic models:
 
+??? Code
+
     ```python
     from pydantic import BaseModel
     from openai import OpenAI
@@ -228,8 +240,6 @@ print("Name:", message.parsed.name)
     print("Age:", message.parsed.age)
     ```
 
-Output:
-
 ```console
 ParsedChatCompletionMessage[Testing](content='{"name": "Cameron", "age": 28}', refusal=None, role='assistant', audio=None, function_call=None, tool_calls=[], parsed=Testing(name='Cameron', age=28))
 Name: Cameron
@@ -238,6 +248,8 @@ Age: 28
 
 Here is a more complex example using nested Pydantic models to handle a step-by-step math solution:
 
+??? Code
+
     ```python
     from typing import List
     from pydantic import BaseModel
@@ -296,6 +308,8 @@ These parameters can be used in the same way as the parameters from the Online
 Serving examples above. One example for the usage of the `choice` parameter is
 shown below:
 
+??? Code
+
     ```python
     from vllm import LLM, SamplingParams
     from vllm.sampling_params import GuidedDecodingParams

docs/features/tool_calling.md

@@ -15,6 +15,8 @@ vllm serve meta-llama/Llama-3.1-8B-Instruct \
 
 Next, make a request to the model that should result in it using the available tools:
 
+??? Code
+
     ```python
     from openai import OpenAI
     import json
@@ -301,6 +303,8 @@ A tool parser plugin is a Python file containing one or more ToolParser implemen
 
 Here is a summary of a plugin file:
 
+??? Code
+
     ```python
 
     # import the required packages

docs/getting_started/installation/cpu.md

@@ -76,6 +76,8 @@ Currently, there are no pre-built CPU wheels.
 
 ### Build image from source
 
+??? Commands
+
     ```console
     $ docker build -f docker/Dockerfile.cpu --tag vllm-cpu-env --target vllm-openai .
 
@@ -144,6 +146,8 @@ vllm serve facebook/opt-125m
 
 - If using vLLM CPU backend on a machine with hyper-threading, it is recommended to bind only one OpenMP thread on each physical CPU core using `VLLM_CPU_OMP_THREADS_BIND` or using auto thread binding feature by default. On a hyper-threading enabled platform with 16 logical CPU cores / 8 physical CPU cores:
 
+??? Commands
+
     ```console
     $ lscpu -e # check the mapping between logical CPU cores and physical CPU cores
 

docs/getting_started/installation/gpu/rocm.inc.md

@@ -90,6 +90,8 @@ Currently, there are no pre-built ROCm wheels.
 
 4. Build vLLM. For example, vLLM on ROCM 6.3 can be built with the following steps:
 
+    ??? Commands
+
         ```bash
         pip install --upgrade pip
 
@@ -201,6 +203,8 @@ DOCKER_BUILDKIT=1 docker build \
 
 To run the above docker image `vllm-rocm`, use the below command:
 
+??? Command
+
     ```console
     docker run -it \
     --network=host \

docs/getting_started/installation/intel_gaudi.md

@@ -200,7 +200,7 @@ INFO 08-01 21:37:59 hpu_model_runner.py:509] Generated 48 decode buckets: [(1, 1
 
 `min` determines the lowest value of the bucket. `step` determines the interval between buckets, and `max` determines the upper bound of the bucket. Furthermore, interval between `min` and `step` has special handling -- `min` gets multiplied by consecutive powers of two, until `step` gets reached. We call this the ramp-up phase and it is used for handling lower batch sizes with minimum wastage, while allowing larger padding on larger batch sizes.
 
-Example (with ramp-up)
+Example (with ramp-up):
 
 ```text
 min = 2, step = 32, max = 64
@@ -209,7 +209,7 @@ min = 2, step = 32, max = 64
 => buckets = ramp_up + stable => (2, 4, 8, 16, 32, 64)
 ```
 
-Example (without ramp-up)
+Example (without ramp-up):
 
 ```text
 min = 128, step = 128, max = 512
@@ -232,6 +232,8 @@ As an example, if a request of 3 sequences, with max sequence length of 412 come
 
 Warmup is an optional, but highly recommended step occurring before vLLM server starts listening. It executes a forward pass for each bucket with dummy data. The goal is to pre-compile all graphs and not incur any graph compilation overheads within bucket boundaries during server runtime. Each warmup step is logged during vLLM startup:
 
+??? Logs
+
     ```text
     INFO 08-01 22:26:47 hpu_model_runner.py:1066] [Warmup][Prompt][1/24] batch_size:4 seq_len:1024 free_mem:79.16 GiB
     INFO 08-01 22:26:47 hpu_model_runner.py:1066] [Warmup][Prompt][2/24] batch_size:4 seq_len:896 free_mem:55.43 GiB
@@ -279,6 +281,8 @@ When there's large amount of requests pending, vLLM scheduler will attempt to fi
 
 Each described step is logged by vLLM server, as follows (negative values correspond to memory being released):
 
+??? Logs
+
     ```text
     INFO 08-02 17:37:44 hpu_model_runner.py:493] Prompt bucket config (min, step, max_warmup) bs:[1, 32, 4], seq:[128, 128, 1024]
     INFO 08-02 17:37:44 hpu_model_runner.py:499] Generated 24 prompt buckets: [(1, 128), (1, 256), (1, 384), (1, 512), (1, 640), (1, 768), (1, 896), (1, 1024), (2, 128), (2, 256), (2, 384), (2, 512), (2, 640), (2, 768), (2, 896), (2, 1024), (4, 128), (4, 256), (4, 384), (4, 512), (4, 640), (4, 768), (4, 896), (4, 1024)]

docs/getting_started/quickstart.md

@@ -147,6 +147,8 @@ curl http://localhost:8000/v1/completions \
 
 Since this server is compatible with OpenAI API, you can use it as a drop-in replacement for any applications using OpenAI API. For example, another way to query the server is via the `openai` Python package:
 
+??? Code
+
     ```python
     from openai import OpenAI
 
@@ -184,6 +186,8 @@ curl http://localhost:8000/v1/chat/completions \
 
 Alternatively, you can use the `openai` Python package:
 
+??? Code
+
     ```python
     from openai import OpenAI
     # Set OpenAI's API key and API base to use vLLM's API server.

docs/models/generative_models.md

@@ -85,6 +85,8 @@ and automatically applies the model's [chat template](https://huggingface.co/doc
     In general, only instruction-tuned models have a chat template.
     Base models may perform poorly as they are not trained to respond to the chat conversation.
 
+??? Code
+
     ```python
     from vllm import LLM
 

docs/models/supported_models.md

@@ -70,7 +70,10 @@ To make your model compatible with the Transformers backend, it needs:
 2. `MyAttention` must use `ALL_ATTENTION_FUNCTIONS` to call attention.
 3. `MyModel` must contain `_supports_attention_backend = True`.
 
-```python title="modeling_my_model.py"
+<details>
+<summary>modeling_my_model.py</summary>
+
+```python
 
 from transformers import PreTrainedModel
 from torch import nn
@@ -93,6 +96,8 @@ class MyModel(PreTrainedModel):
     _supports_attention_backend = True
 ```
 
+</details>
+
 Here is what happens in the background when this model is loaded:
 
 1. The config is loaded.
@@ -103,7 +108,10 @@ That's it!
 
 For your model to be compatible with vLLM's tensor parallel and/or pipeline parallel features, you must add `base_model_tp_plan` and/or `base_model_pp_plan` to your model's config class:
 
-```python title="configuration_my_model.py"
+<details>
+<summary>configuration_my_model.py</summary>
+
+```python
 
 from transformers import PretrainedConfig
 
@@ -123,6 +131,8 @@ class MyConfig(PretrainedConfig):
     }
 ```
 
+</details>
+
 - `base_model_tp_plan` is a `dict` that maps fully qualified layer name patterns to tensor parallel styles (currently only `"colwise"` and `"rowwise"` are supported).
 - `base_model_pp_plan` is a `dict` that maps direct child layer names to `tuple`s of `list`s of `str`s:
     * You only need to do this for layers which are not present on all pipeline stages
@@ -198,6 +208,9 @@ huggingface-cli scan-cache --dir ~/.cache/huggingface/hub
 
 Use the Hugging Face CLI to interactively [delete downloaded model](https://huggingface.co/docs/huggingface_hub/guides/manage-cache#clean-your-cache) from the cache:
 
+<details>
+<summary>Commands</summary>
+
 ```console
 # The `delete-cache` command requires extra dependencies to work with the TUI.
 # Please run `pip install huggingface_hub[cli]` to install them.
@@ -224,6 +237,8 @@ Start deletion.
 Done. Deleted 1 repo(s) and 0 revision(s) for a total of 438.9M.
 ```
 
+</details>
+
 #### Using a proxy
 
 Here are some tips for loading/downloading models from Hugging Face using a proxy:
@@ -601,6 +616,8 @@ Specified using `--task generate`.
 
     For the best results, we recommend using the following dependency versions (tested on A10 and L40):
 
+    ??? Dependency versions
+
         ```text
         # Core vLLM-compatible dependencies with Molmo accuracy setup (tested on L40)
         torch==2.5.1

docs/serving/integrations/langchain.md

@@ -13,6 +13,8 @@ pip install langchain langchain_community -q
 
 To run inference on a single or multiple GPUs, use `VLLM` class from `langchain`.
 
+??? Code
+
     ```python
     from langchain_community.llms import VLLM
 

docs/serving/openai_compatible_server.md

@@ -15,6 +15,8 @@ vllm serve NousResearch/Meta-Llama-3-8B-Instruct \
 
 To call the server, in your preferred text editor, create a script that uses an HTTP client. Include any messages that you want to send to the model. Then run that script. Below is an example script using the [official OpenAI Python client](https://github.com/openai/openai-python).
 
+??? Code
+
     ```python
     from openai import OpenAI
     client = OpenAI(
@@ -147,6 +149,8 @@ with `--enable-request-id-headers`.
 > rather than within the vLLM layer for this reason.
 > See [this PR](https://github.com/vllm-project/vllm/pull/11529) for more details.
 
+??? Code
+
     ```python
     completion = client.chat.completions.create(
         model="NousResearch/Meta-Llama-3-8B-Instruct",
@@ -184,12 +188,16 @@ Code example: <gh-file:examples/online_serving/openai_completion_client.py>
 
 The following [sampling parameters][sampling-params] are supported.
 
+??? Code
+
     ```python
     --8<-- "vllm/entrypoints/openai/protocol.py:completion-sampling-params"
     ```
 
 The following extra parameters are supported:
 
+??? Code
+
     ```python
     --8<-- "vllm/entrypoints/openai/protocol.py:completion-extra-params"
     ```
@@ -212,12 +220,16 @@ Code example: <gh-file:examples/online_serving/openai_chat_completion_client.py>
 
 The following [sampling parameters][sampling-params] are supported.
 
+??? Code
+
     ```python
     --8<-- "vllm/entrypoints/openai/protocol.py:chat-completion-sampling-params"
     ```
 
 The following extra parameters are supported:
 
+??? Code
+
     ```python
     --8<-- "vllm/entrypoints/openai/protocol.py:chat-completion-extra-params"
     ```
@@ -259,6 +271,8 @@ and passing a list of `messages` in the request. Refer to the examples below for
 
     Since the request schema is not defined by OpenAI client, we post a request to the server using the lower-level `requests` library:
 
+    ??? Code
+
         ```python
         import requests
 
@@ -316,12 +330,16 @@ The following [pooling parameters][pooling-params] are supported.
 
 The following extra parameters are supported by default:
 
+??? Code
+
     ```python
     --8<-- "vllm/entrypoints/openai/protocol.py:embedding-extra-params"
     ```
 
 For chat-like input (i.e. if `messages` is passed), these extra parameters are supported instead:
 
+??? Code
+
     ```python
     --8<-- "vllm/entrypoints/openai/protocol.py:chat-embedding-extra-params"
     ```
@@ -343,12 +361,16 @@ Code example: <gh-file:examples/online_serving/openai_transcription_client.py>
 
 The following [sampling parameters][sampling-params] are supported.
 
+??? Code
+
     ```python
     --8<-- "vllm/entrypoints/openai/protocol.py:transcription-sampling-params"
     ```
 
 The following extra parameters are supported:
 
+??? Code
+
     ```python
     --8<-- "vllm/entrypoints/openai/protocol.py:transcription-extra-params"
     ```
@@ -387,8 +409,6 @@ Code example: <gh-file:examples/online_serving/openai_classification_client.py>
 
 You can classify multiple texts by passing an array of strings:
 
-Request:
-
 ```bash
 curl -v "http://127.0.0.1:8000/classify" \
   -H "Content-Type: application/json" \
@@ -401,7 +421,7 @@ curl -v "http://127.0.0.1:8000/classify" \
   }'
 ```
 
-Response:
+??? Response
 
     ```bash
     {
@@ -440,8 +460,6 @@ Response:
 
 You can also pass a string directly to the `input` field:
 
-Request:
-
 ```bash
 curl -v "http://127.0.0.1:8000/classify" \
   -H "Content-Type: application/json" \
@@ -451,7 +469,7 @@ curl -v "http://127.0.0.1:8000/classify" \
   }'
 ```
 
-Response:
+??? Response
 
     ```bash
     {
@@ -508,8 +526,6 @@ Code example: <gh-file:examples/online_serving/openai_cross_encoder_score.py>
 
 You can pass a string to both `text_1` and `text_2`, forming a single sentence pair.
 
-Request:
-
 ```bash
 curl -X 'POST' \
   'http://127.0.0.1:8000/score' \
@@ -523,7 +539,7 @@ curl -X 'POST' \
 }'
 ```
 
-Response:
+??? Response
 
     ```bash
     {
@@ -548,7 +564,7 @@ You can pass a string to `text_1` and a list to `text_2`, forming multiple sente
 where each pair is built from `text_1` and a string in `text_2`.
 The total number of pairs is `len(text_2)`.
 
-Request:
+??? Request
 
     ```bash
     curl -X 'POST' \
@@ -565,7 +581,7 @@ curl -X 'POST' \
     }'
     ```
 
-Response:
+??? Response
 
     ```bash
     {
@@ -593,7 +609,7 @@ You can pass a list to both `text_1` and `text_2`, forming multiple sentence pai
 where each pair is built from a string in `text_1` and the corresponding string in `text_2` (similar to `zip()`).
 The total number of pairs is `len(text_2)`.
 
-Request:
+??? Request
 
     ```bash
     curl -X 'POST' \
@@ -614,7 +630,7 @@ curl -X 'POST' \
     }'
     ```
 
-Response:
+??? Response
 
     ```bash
     {
@@ -675,7 +691,7 @@ Code example: <gh-file:examples/online_serving/jinaai_rerank_client.py>
 Note that the `top_n` request parameter is optional and will default to the length of the `documents` field.
 Result documents will be sorted by relevance, and the `index` property can be used to determine original order.
 
-Request:
+??? Request
 
     ```bash
     curl -X 'POST' \
@@ -693,7 +709,7 @@ curl -X 'POST' \
     }'
     ```
 
-Response:
+??? Response
 
     ```bash
     {

docs/usage/metrics.md

@@ -12,6 +12,8 @@ vllm serve unsloth/Llama-3.2-1B-Instruct
 
 Then query the endpoint to get the latest metrics from the server:
 
+??? Output
+
     ```console
     $ curl http://0.0.0.0:8000/metrics
 
@@ -31,6 +33,8 @@ vllm:iteration_tokens_total_bucket{le="512.0",model_name="unsloth/Llama-3.2-1B-I
 
 The following metrics are exposed:
 
+??? Code
+
     ```python
     --8<-- "vllm/engine/metrics.py:metrics-definitions"
     ```

docs/usage/troubleshooting.md

@@ -60,6 +60,8 @@ To identify the particular CUDA operation that causes the error, you can add `--
 
 If GPU/CPU communication cannot be established, you can use the following Python script and follow the instructions below to confirm whether the GPU/CPU communication is working correctly.
 
+??? Code
+
     ```python
     # Test PyTorch NCCL
     import torch
@@ -165,6 +167,8 @@ WARNING 12-11 14:50:37 multiproc_worker_utils.py:281] CUDA was previously
 
 or an error from Python that looks like this:
 
+??? Logs
+
     ```console
     RuntimeError:
             An attempt has been made to start a new process before the
@@ -207,6 +211,8 @@ if __name__ == '__main__':
 
 vLLM heavily depends on `torch.compile` to optimize the model for better performance, which introduces the dependency on the `torch.compile` functionality and the `triton` library. By default, we use `torch.compile` to [optimize some functions](https://github.com/vllm-project/vllm/pull/10406) in the model. Before running vLLM, you can check if `torch.compile` is working as expected by running the following script:
 
+??? Code
+
     ```python
     import torch
 

docs/usage/usage_stats.md

@@ -10,6 +10,8 @@ The list of data collected by the latest version of vLLM can be found here: <gh-
 
 Here is an example as of v0.4.0:
 
+??? Output
+
     ```json
     {
       "uuid": "fbe880e9-084d-4cab-a395-8984c50f1109",