mirror of
https://git.datalinker.icu/vllm-project/vllm.git
synced 2026-01-01 12:21:50 +08:00
a3a73ab069
The 2nd PR for #4532. This PR supports loading FP8 kv-cache scaling factors from a FP8 checkpoint (with .kv_scale parameter).
Benchmarking vLLM
Downloading the ShareGPT dataset
You can download the dataset by running:
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json