xinyun/vllm
1
0
Fork 0
mirror of https://git.datalinker.icu/vllm-project/vllm.git synced 2026-05-31 06:37:04 +08:00
Code Issues Packages Projects Releases Wiki Activity

[Docs] Improve docstring for ray data llm example (#20597)

Browse Source 
Signed-off-by: Ricardo Decal <rdecal@anyscale.com>
This commit is contained in:
Ricardo Decal 2025-07-07 20:06:26 -07:00 committed by GitHub
parent 0d914c81a2
commit e60d422f19
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 11 additions and 9 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

20
examples/offline_inference/batch_llm_inference.py
View File

@ -3,17 +3,19 @@
""" """
This example shows how to use Ray Data for data parallel batch inference. This example shows how to use Ray Data for data parallel batch inference.
Ray Data is a data processing framework that can handle large datasets Ray Data is a data processing framework that can process very large datasets
and integrates tightly with vLLM for data-parallel inference. with first-class support for vLLM.
As of Ray 2.44, Ray Data has a native integration with
vLLM (under ray.data.llm).
Ray Data provides functionality for: Ray Data provides functionality for:
* Reading and writing to cloud storage (S3, GCS, etc.) * Reading and writing to most popular file formats and cloud object storage.
* Automatic sharding and load-balancing across a cluster * Streaming execution, so you can run inference on datasets that far exceed
* Optimized configuration of vLLM using continuous batching the aggregate RAM of the cluster.
* Compatible with tensor/pipeline parallel inference as well. * Scale up the workload without code changes.
* Automatic sharding, load-balancing, and autoscaling across a Ray cluster,
with built-in fault-tolerance and retry semantics.
* Continuous batching that keeps vLLM replicas saturated and maximizes GPU
utilization.
* Compatible with tensor/pipeline parallel inference.
Learn more about Ray Data's LLM integration: Learn more about Ray Data's LLM integration:
https://docs.ray.io/en/latest/data/working-with-llms.html https://docs.ray.io/en/latest/data/working-with-llms.html