mirror of https://git.datalinker.icu/vllm-project/vllm.git synced 2025-12-15 09:55:57 +08:00

History

[Core] Implement disagg prefill by StatelessProcessGroup (#10502 )

This PR provides initial support for single-node disaggregated prefill in 1P1D scenario.
Signed-off-by: KuntaiDu <kuntai@uchicago.edu>
Co-authored-by: ApostaC <yihua98@uchicago.edu>
Co-authored-by: YaoJiayi <120040070@link.cuhk.edu.cn>

2024-12-01 19:01:00 -06:00

cutlass_benchmarks

[Kernel] Add per-tensor and per-token AZP epilogues (#5941 )

2024-08-06 18:17:08 +00:00

disagg_benchmarks

[Core] Implement disagg prefill by StatelessProcessGroup (#10502 )

2024-12-01 19:01:00 -06:00

kernels

[Model][Quantization] HQQ support through Marlin kernel expansion (#9766 )

2024-11-19 13:31:12 -08:00

overheads

[Core] Deprecating block manager v1 and make block manager v2 default (#8704 )

2024-10-17 11:38:15 -05:00

backend_request_func.py

fix the issue that len(tokenizer(prompt)["input_ids"]) > prompt_len (#10524 )

2024-11-21 11:15:36 +00:00

benchmark_latency.py

[Misc] Make benchmarks use EngineArgs (#9529 )

2024-10-22 15:40:38 -07:00

benchmark_prefix_caching.py

[misc] partial prefix & random input generation benchmark (#9929 )

2024-11-18 15:39:14 -08:00

benchmark_prioritization.py

[Misc] Make benchmarks use EngineArgs (#9529 )

2024-10-22 15:40:38 -07:00

benchmark_serving.py

[Misc] Adding MMMU-Pro vision dataset to serving benchmark (#10804 )

2024-12-01 08:47:05 +00:00

benchmark_throughput.py

[Feature] Update benchmark_throughput.py to support image input (#9851 )

2024-11-05 19:30:02 +00:00

launch_tgi_server.sh

[CI/Build] Add shell script linting using shellcheck (#7925 )

2024-11-07 18:17:29 +00:00

README.md

[Feature] Update benchmark_throughput.py to support image input (#9851 )

2024-11-05 19:30:02 +00:00

sonnet.txt

feat(benchmarks): Add Prefix Caching Benchmark to Serving Benchmark (#3277 )

2024-03-27 13:39:26 -07:00

README.md

Benchmarking vLLM

Downloading the ShareGPT dataset

You can download the dataset by running:

wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

Downloading the ShareGPT4V dataset

The json file refers to several image datasets (coco, llava, etc.). The benchmark scripts will ignore a datapoint if the referred image is missing.

wget https://huggingface.co/datasets/Lin-Chen/ShareGPT4V/resolve/main/sharegpt4v_instruct_gpt4-vision_cap100k.json
mkdir coco -p
wget http://images.cocodataset.org/zips/train2017.zip -O coco/train2017.zip
unzip coco/train2017.zip -d coco/