xinyun/vllm
1
0
Fork 0
mirror of https://git.datalinker.icu/vllm-project/vllm.git synced 2025-12-14 08:44:58 +08:00
Code Issues Packages Projects Releases Wiki Activity

[Doc] Update OOT model docs (#18742)

Browse Source 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
This commit is contained in:
Cyrus Leung 2025-05-27 14:30:31 +08:00 committed by GitHub
parent d260f799a9
commit 25a817f202
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
2 changed files with 18 additions and 17 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

29
docs/contributing/model/registration.md
View File

@ -23,33 +23,32 @@ Finally, update our [list of supported models][supported-models] to promote your
## Out-of-tree models ## Out-of-tree models
You can load an external model using a plugin without modifying the vLLM codebase. You can load an external model [using a plugin][plugin-system] without modifying the vLLM codebase.
!!! info
[vLLM's Plugin System][plugin-system]
To register the model, use the following code: To register the model, use the following code:
```python ```python
from vllm import ModelRegistry # The entrypoint of your plugin
from your_code import YourModelForCausalLM def register():
ModelRegistry.register_model("YourModelForCausalLM", YourModelForCausalLM) from vllm import ModelRegistry
from your_code import YourModelForCausalLM
ModelRegistry.register_model("YourModelForCausalLM", YourModelForCausalLM)
``` ```
If your model imports modules that initialize CUDA, consider lazy-importing it to avoid errors like `RuntimeError: Cannot re-initialize CUDA in forked subprocess`: If your model imports modules that initialize CUDA, consider lazy-importing it to avoid errors like `RuntimeError: Cannot re-initialize CUDA in forked subprocess`:
```python ```python
from vllm import ModelRegistry # The entrypoint of your plugin
def register():
from vllm import ModelRegistry
ModelRegistry.register_model( ModelRegistry.register_model(
"YourModelForCausalLM", "YourModelForCausalLM",
"your_code:YourModelForCausalLM" "your_code:YourModelForCausalLM"
) )
``` ```
!!! warning !!! warning
If your model is a multimodal model, ensure the model class implements the [SupportsMultiModal][vllm.model_executor.models.interfaces.SupportsMultiModal] interface. If your model is a multimodal model, ensure the model class implements the [SupportsMultiModal][vllm.model_executor.models.interfaces.SupportsMultiModal] interface.
Read more about that [here][supports-multimodal]. Read more about that [here][supports-multimodal].
!!! note
Although you can directly put these code snippets in your script using `vllm.LLM`, the recommended way is to place these snippets in a vLLM plugin. This ensures compatibility with various vLLM features like distributed inference and the API server.

6
docs/design/plugin_system.md
View File

@ -30,8 +30,10 @@ def register():
from vllm import ModelRegistry from vllm import ModelRegistry
if "MyLlava" not in ModelRegistry.get_supported_archs(): if "MyLlava" not in ModelRegistry.get_supported_archs():
ModelRegistry.register_model("MyLlava", ModelRegistry.register_model(
"vllm_add_dummy_model.my_llava:MyLlava") "MyLlava",
"vllm_add_dummy_model.my_llava:MyLlava",
)
``` ```
For more information on adding entry points to your package, please check the [official documentation](https://setuptools.pypa.io/en/latest/userguide/entry_point.html). For more information on adding entry points to your package, please check the [official documentation](https://setuptools.pypa.io/en/latest/userguide/entry_point.html).