xinyun/vllm
1
0
Fork 0
mirror of https://git.datalinker.icu/vllm-project/vllm.git synced 2026-03-24 04:16:52 +08:00
Code Issues Packages Projects Releases Wiki Activity

[Docs] Add Ray Serve LLM section to openai compatible server guide (#20595)

Browse Source 
Signed-off-by: Ricardo Decal <rdecal@anyscale.com>
This commit is contained in:
Ricardo Decal 2025-07-07 20:08:05 -07:00 committed by GitHub
parent e60d422f19
commit 90a2769f20
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 14 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
docs/serving/openai_compatible_server.md
View File

@ -775,3 +775,17 @@ The following extra parameters are supported:
```python
--8<-- "vllm/entrypoints/openai/protocol.py:rerank-extra-params"
```
## Ray Serve LLM
Ray Serve LLM enables scalable, production-grade serving of the vLLM engine. It integrates tightly with vLLM and extends it with features such as auto-scaling, load balancing, and back-pressure.
Key capabilities:
- Exposes an OpenAI-compatible HTTP API as well as a Pythonic API.
- Scales from a single GPU to a multi-node cluster without code changes.
- Provides observability and autoscaling policies through Ray dashboards and metrics.
The following example shows how to deploy a large model like DeepSeek R1 with Ray Serve LLM: <gh-file:examples/online_serving/ray_serve_deepseek.py>.
Learn more about Ray Serve LLM with the official [Ray Serve LLM documentation](https://docs.ray.io/en/latest/serve/llm/serving-llms.html).