xinyun/vllm
1
0
Fork 0
mirror of https://git.datalinker.icu/vllm-project/vllm.git synced 2025-12-10 08:04:58 +08:00
Code Issues Packages Projects Releases Wiki Activity

docs: Add BentoML deployment doc (#3336)

Browse Source 
Signed-off-by: Sherlock113 <sherlockxu07@gmail.com>
This commit is contained in:
Sherlock Xu 2024-03-13 01:34:30 +08:00 committed by GitHub
parent 654865e21d
commit b0925b3878
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
2 changed files with 9 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
docs/source/index.rst
View File

@ -73,6 +73,7 @@ Documentation
serving/run_on_sky
serving/deploying_with_kserve
serving/deploying_with_triton
serving/deploying_with_bentoml
serving/deploying_with_docker
serving/serving_with_langchain
serving/metrics

8
docs/source/serving/deploying_with_bentoml.rst Normal file
View File

@ -0,0 +1,8 @@
.. _deploying_with_bentoml:
Deploying with BentoML
======================
`BentoML <https://github.com/bentoml/BentoML>`_ allows you to deploy a large language model (LLM) server with vLLM as the backend, which exposes OpenAI-compatible endpoints. You can serve the model locally or containerize it as an OCI-complicant image and deploy it on Kubernetes.
For details, see the tutorial `vLLM inference in the BentoML documentation <https://docs.bentoml.com/en/latest/use-cases/large-language-models/vllm.html>`_.