xinyun/vllm
1
0
Fork 0
mirror of https://git.datalinker.icu/vllm-project/vllm.git synced 2025-12-09 04:45:01 +08:00
Code Issues Packages Projects Releases Wiki Activity

[Docs] Revise frameworks/anything-llm.md (#24489)

Browse Source 
Signed-off-by: windsonsea <haifeng.yao@daocloud.io>
This commit is contained in:
Michael Yao 2025-09-11 16:50:05 +08:00 committed by GitHub
parent 6aeb1dab4a
commit 85df8afdae
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 34 additions and 22 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

56
docs/deployment/frameworks/anything-llm.md
View File

@ -1,41 +1,53 @@
# Anything LLM
# AnythingLLM
[Anything LLM](https://github.com/Mintplex-Labs/anything-llm) is a full-stack application that enables you to turn any document, resource, or piece of content into context that any LLM can use as references during chatting.
[AnythingLLM](https://github.com/Mintplex-Labs/anything-llm) is a full-stack application that enables you to turn any document, resource, or piece of content into context that any LLM can use as references during chatting.
It allows you to deploy a large language model (LLM) server with vLLM as the backend, which exposes OpenAI-compatible endpoints.
## Prerequisites
- Setup vLLM environment
Set up the vLLM environment:
```bash
pip install vllm
```
## Deploy
- Start the vLLM server with the supported chat completion model, e.g.
1. Start the vLLM server with a supported chat-completion model, for example:
```bash
vllm serve Qwen/Qwen1.5-32B-Chat-AWQ --max-model-len 4096
```
```bash
vllm serve Qwen/Qwen1.5-32B-Chat-AWQ --max-model-len 4096
```
- Download and install [Anything LLM desktop](https://anythingllm.com/desktop).
1. Download and install [AnythingLLM Desktop](https://anythingllm.com/desktop).
- On the bottom left of open settings, AI Providers --> LLM:
- LLM Provider: Generic OpenAI
- Base URL: http://{vllm server host}:{vllm server port}/v1
- Chat Model Name: `Qwen/Qwen1.5-32B-Chat-AWQ`
1. Configure the AI provider:
![](../../assets/deployment/anything-llm-provider.png)
- At the bottom, click the 🔧 wrench icon -> **Open settings** -> **AI Providers** -> **LLM**.
- Enter the following values:
- LLM Provider: Generic OpenAI
- Base URL: `http://{vllm server host}:{vllm server port}/v1`
- Chat Model Name: `Qwen/Qwen1.5-32B-Chat-AWQ`
- Back to home page, New Workspace --> create `vllm` workspace, and start to chat:
![set AI providers](../../assets/deployment/anything-llm-provider.png)
![](../../assets/deployment/anything-llm-chat-without-doc.png)
1. Create a workspace:
- Click the upload button:
- upload the doc
- select the doc and move to the workspace
- save and embed
1. At the bottom, click the ↺ back icon and back to workspaces.
1. Create a workspace (e.g., `vllm`) and start chatting.
![](../../assets/deployment/anything-llm-upload-doc.png)
![create a workspace](../../assets/deployment/anything-llm-chat-without-doc.png)
- Chat again:
1. Add a document.
![](../../assets/deployment/anything-llm-chat-with-doc.png)
1. Click the 📎 attachment icon.
1. Upload a document.
1. Select and move the document into your workspace.
1. Save and embed it.
![add a document](../../assets/deployment/anything-llm-upload-doc.png)
1. Chat using your document as context.
![chat with your context](../../assets/deployment/anything-llm-chat-with-doc.png)