xinyun/vllm
1
0
Fork 0
mirror of https://git.datalinker.icu/vllm-project/vllm.git synced 2025-12-10 05:34:57 +08:00
Code Issues Packages Projects Releases Wiki Activity

[Doc] Update installation doc (#3746)

Browse Source 
[Doc] Update installation doc for build from source and explain the dependency on torch/cuda version (#3746)
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
This commit is contained in:
youkaichao 2024-03-30 16:34:38 -07:00 committed by GitHub
parent b6d103542c
commit 9c82a1bec3
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 21 additions and 12 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

33
docs/source/getting_started/installation.rst
View File

@ -19,7 +19,7 @@ You can install vLLM using pip:
.. code-block:: console .. code-block:: console
$ # (Optional) Create a new conda environment. $ # (Recommended) Create a new conda environment.
$ conda create -n myenv python=3.9 -y $ conda create -n myenv python=3.9 -y
$ conda activate myenv $ conda activate myenv
@ -28,24 +28,19 @@ You can install vLLM using pip:
.. note:: .. note::
As of now, vLLM's binaries are compiled on CUDA 12.1 by default. As of now, vLLM's binaries are compiled with CUDA 12.1 and public PyTorch release versions by default.
However, you can install vLLM with CUDA 11.8 by running: We also provide vLLM binaries compiled with CUDA 11.8 and public PyTorch release versions:
.. code-block:: console .. code-block:: console
$ # Install vLLM with CUDA 11.8. $ # Install vLLM with CUDA 11.8.
$ export VLLM_VERSION=0.2.4 $ export VLLM_VERSION=0.4.0
$ export PYTHON_VERSION=39 $ export PYTHON_VERSION=39
$ pip install https://github.com/vllm-project/vllm/releases/download/v${VLLM_VERSION}/vllm-${VLLM_VERSION}+cu118-cp${PYTHON_VERSION}-cp${PYTHON_VERSION}-manylinux1_x86_64.whl $ pip install https://github.com/vllm-project/vllm/releases/download/v${VLLM_VERSION}/vllm-${VLLM_VERSION}+cu118-cp${PYTHON_VERSION}-cp${PYTHON_VERSION}-manylinux1_x86_64.whl --extra-index-url https://download.pytorch.org/whl/cu118
$ # Re-install PyTorch with CUDA 11.8. In order to be performant, vLLM has to compile many cuda kernels. The compilation unfortunately introduces binary incompatibility with other CUDA versions and PyTorch versions, even for the same PyTorch version with different building configurations.
$ pip uninstall torch -y
$ pip install torch --upgrade --index-url https://download.pytorch.org/whl/cu118
$ # Re-install xFormers with CUDA 11.8.
$ pip uninstall xformers -y
$ pip install --upgrade xformers --index-url https://download.pytorch.org/whl/cu118
Therefore, it is recommended to install vLLM with a **fresh new** conda environment. If either you have a different CUDA version or you want to use an existing PyTorch installation, you need to build vLLM from source. See below for instructions.
.. _build_from_source: .. _build_from_source:
@ -77,6 +72,20 @@ You can also build and install vLLM from source:
$ # Use `--ipc=host` to make sure the shared memory is large enough. $ # Use `--ipc=host` to make sure the shared memory is large enough.
$ docker run --gpus all -it --rm --ipc=host nvcr.io/nvidia/pytorch:23.10-py3 $ docker run --gpus all -it --rm --ipc=host nvcr.io/nvidia/pytorch:23.10-py3
If you don't want to use docker, it is recommended to have a full installation of CUDA Toolkit. You can download and install it from `the official website <https://developer.nvidia.com/cuda-toolkit-archive>`_. After installation, set the environment variable `CUDA_HOME` to the installation path of CUDA Toolkit, and make sure that the `nvcc` compiler is in your `PATH`, e.g.:
.. code-block:: console
$ export CUDA_HOME=/usr/local/cuda
$ export PATH="${CUDA_HOME}/bin:$PATH"
Here is a sanity check to verify that the CUDA Toolkit is correctly installed:
.. code-block:: console
$ nvcc --version # verify that nvcc is in your PATH
$ ${CUDA_HOME}/bin/nvcc --version # verify that nvcc is in your CUDA_HOME
.. note:: .. note::
If you are developing the C++ backend of vLLM, consider building vLLM with If you are developing the C++ backend of vLLM, consider building vLLM with