mirror of
https://git.datalinker.icu/vllm-project/vllm.git
synced 2025-12-10 03:15:20 +08:00
[Doc] Move additional tips/notes to the top (#9647)
This commit is contained in:
parent
ad6f78053e
commit
b979143d5b
@ -3,10 +3,47 @@
|
|||||||
Supported Models
|
Supported Models
|
||||||
================
|
================
|
||||||
|
|
||||||
vLLM supports a variety of generative Transformer models in `HuggingFace (HF) Transformers <https://huggingface.co/models>`_.
|
vLLM supports a variety of generative and embedding models from `HuggingFace (HF) Transformers <https://huggingface.co/models>`_.
|
||||||
The following is the list of model architectures that are currently supported by vLLM.
|
This page lists the model architectures that are currently supported by vLLM.
|
||||||
Alongside each architecture, we include some popular models that use it.
|
Alongside each architecture, we include some popular models that use it.
|
||||||
|
|
||||||
|
For other models, you can check the :code:`config.json` file inside the model repository.
|
||||||
|
If the :code:`"architectures"` field contains a model architecture listed below, then it should be supported in theory.
|
||||||
|
|
||||||
|
.. tip::
|
||||||
|
The easiest way to check if your model is really supported at runtime is to run the program below:
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
from vllm import LLM
|
||||||
|
|
||||||
|
llm = LLM(model=...) # Name or path of your model
|
||||||
|
output = llm.generate("Hello, my name is")
|
||||||
|
print(output)
|
||||||
|
|
||||||
|
If vLLM successfully generates text, it indicates that your model is supported.
|
||||||
|
|
||||||
|
Otherwise, please refer to :ref:`Adding a New Model <adding_a_new_model>` and :ref:`Enabling Multimodal Inputs <enabling_multimodal_inputs>`
|
||||||
|
for instructions on how to implement your model in vLLM.
|
||||||
|
Alternatively, you can `open an issue on GitHub <https://github.com/vllm-project/vllm/issues/new/choose>`_ to request vLLM support.
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
To use models from `ModelScope <https://www.modelscope.cn>`_ instead of HuggingFace Hub, set an environment variable:
|
||||||
|
|
||||||
|
.. code-block:: shell
|
||||||
|
|
||||||
|
$ export VLLM_USE_MODELSCOPE=True
|
||||||
|
|
||||||
|
And use with :code:`trust_remote_code=True`.
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
from vllm import LLM
|
||||||
|
|
||||||
|
llm = LLM(model=..., revision=..., trust_remote_code=True) # Name or path of your model
|
||||||
|
output = llm.generate("Hello, my name is")
|
||||||
|
print(output)
|
||||||
|
|
||||||
Text-only Language Models
|
Text-only Language Models
|
||||||
^^^^^^^^^^^^^^^^^^^^^^^^^
|
^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
@ -515,44 +552,6 @@ Multimodal Embedding
|
|||||||
Some model architectures support both generation and embedding tasks.
|
Some model architectures support both generation and embedding tasks.
|
||||||
In this case, you have to pass :code:`--task embedding` to run the model in embedding mode.
|
In this case, you have to pass :code:`--task embedding` to run the model in embedding mode.
|
||||||
|
|
||||||
----
|
|
||||||
|
|
||||||
If your model uses one of the above model architectures, you can seamlessly run your model with vLLM.
|
|
||||||
Otherwise, please refer to :ref:`Adding a New Model <adding_a_new_model>` and :ref:`Enabling Multimodal Inputs <enabling_multimodal_inputs>`
|
|
||||||
for instructions on how to implement support for your model.
|
|
||||||
Alternatively, you can raise an issue on our `GitHub <https://github.com/vllm-project/vllm/issues>`_ project.
|
|
||||||
|
|
||||||
.. tip::
|
|
||||||
The easiest way to check if your model is supported is to run the program below:
|
|
||||||
|
|
||||||
.. code-block:: python
|
|
||||||
|
|
||||||
from vllm import LLM
|
|
||||||
|
|
||||||
llm = LLM(model=...) # Name or path of your model
|
|
||||||
output = llm.generate("Hello, my name is")
|
|
||||||
print(output)
|
|
||||||
|
|
||||||
If vLLM successfully generates text, it indicates that your model is supported.
|
|
||||||
|
|
||||||
.. tip::
|
|
||||||
To use models from `ModelScope <https://www.modelscope.cn>`_ instead of HuggingFace Hub, set an environment variable:
|
|
||||||
|
|
||||||
.. code-block:: shell
|
|
||||||
|
|
||||||
$ export VLLM_USE_MODELSCOPE=True
|
|
||||||
|
|
||||||
And use with :code:`trust_remote_code=True`.
|
|
||||||
|
|
||||||
.. code-block:: python
|
|
||||||
|
|
||||||
from vllm import LLM
|
|
||||||
|
|
||||||
llm = LLM(model=..., revision=..., trust_remote_code=True) # Name or path of your model
|
|
||||||
output = llm.generate("Hello, my name is")
|
|
||||||
print(output)
|
|
||||||
|
|
||||||
|
|
||||||
Model Support Policy
|
Model Support Policy
|
||||||
=====================
|
=====================
|
||||||
|
|
||||||
|
|||||||
Loading…
x
Reference in New Issue
Block a user