xinyun/vllm
1
0
Fork 0
mirror of https://git.datalinker.icu/vllm-project/vllm.git synced 2025-12-09 05:24:54 +08:00
Code Issues Packages Projects Releases Wiki Activity
85 Commits 155 Branches 89 Tags
Go to file
Clone
Open with VS Code Open with VSCodium Open with Intellij IDEA
Download ZIP Download TAR.GZ Download BUNDLE
Woosuk Kwon d359cda5fa Minor
2023-03-26 08:00:39 +00:00
cacheflow
Minor
2023-03-26 08:00:39 +00:00
csrc
Add miscellaneous updates (#8)
2023-03-13 13:48:38 -07:00
tests/kernels
Use FlashAttention for multi_query_kv_attention (#4)
2023-03-01 21:13:08 -08:00
.gitignore
Add gitignore
2023-02-16 07:47:21 +00:00
README.md
Support tensor parallel (#2)
2023-03-21 13:45:42 -07:00
server.py
Support tensor parallel (#2)
2023-03-21 13:45:42 -07:00
setup.py
Implement single_query_cached_kv_attention kernel (#3)
2023-03-01 15:02:19 -08:00

README.md

CacheFlow

Installation

pip install psutil numpy torch transformers
pip install flash-attn # This may take up to 10 mins.
pip install -e .

Run

ray start --head
python server.py [--tensor-parallel-size <N>]
Description
No description provided
Readme 782 MiB
Languages
Python 85.6%
Cuda 8.3%
C++ 4.4%
Shell 0.8%
C 0.4%
Other 0.4%