xinyun/vllm
1
0
Fork 0
mirror of https://git.datalinker.icu/vllm-project/vllm.git synced 2025-12-11 09:06:01 +08:00
Code Issues Packages Projects Releases Wiki Activity

[Docs] Update PP docs (#6598)

Browse Source
This commit is contained in:
Murali Andoorveedu 2024-07-19 19:38:21 -04:00 committed by GitHub
parent 4cc24f01b1
commit 45ceb85a0c
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 2 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
docs/source/serving/distributed_serving.rst
View File

@ -44,11 +44,10 @@ You can also additionally specify :code:`--pipeline-parallel-size` to enable pip
$ vllm serve gpt2 \ $ vllm serve gpt2 \
$ --tensor-parallel-size 4 \ $ --tensor-parallel-size 4 \
$ --pipeline-parallel-size 2 \ $ --pipeline-parallel-size 2
$ --distributed-executor-backend ray
.. note:: .. note::
Pipeline parallel is a beta feature. It is only supported for online serving and the ray backend for now, as well as LLaMa and GPT2 style models. Pipeline parallel is a beta feature. It is only supported for online serving as well as LLaMa, GPT2, and Mixtral style models.
To scale vLLM beyond a single machine, install and start a `Ray runtime <https://docs.ray.io/en/latest/ray-core/starting-ray.html>`_ via CLI before running vLLM: To scale vLLM beyond a single machine, install and start a `Ray runtime <https://docs.ray.io/en/latest/ray-core/starting-ray.html>`_ via CLI before running vLLM: