xinyun/vllm
1
0
Fork 0
mirror of https://git.datalinker.icu/vllm-project/vllm.git synced 2025-12-10 04:26:00 +08:00
Code Issues Packages Projects Releases Wiki Activity

[Docs] Update documentation for gpu-memory-utilization option (#2162)

Browse Source
This commit is contained in:
Suhong Moon 2023-12-17 13:51:57 -05:00 committed by GitHub
parent 671af2b1c0
commit 3ec8c25cd0
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 4 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
docs/source/models/engine_args.rst
View File

@ -89,9 +89,11 @@ Below, you can find an explanation of every engine argument for vLLM:
CPU swap space size (GiB) per GPU.
.. option:: --gpu-memory-utilization <percentage>
.. option:: --gpu-memory-utilization <fraction>
The percentage of GPU memory to be used for the model executor.
The fraction of GPU memory to be used for the model executor, which can range from 0 to 1.
For example, a value of 0.5 would imply 50% GPU memory utilization.
If unspecified, will use the default value of 0.9.
.. option:: --max-num-batched-tokens <tokens>