xinyun/vllm
1
0
Fork 0
mirror of https://git.datalinker.icu/vllm-project/vllm.git synced 2026-04-23 04:07:03 +08:00
Code Issues Packages Projects Releases Wiki Activity
10,300 Commits 155 Branches 89 Tags
Commit Graph

127 Commits

Author SHA1 Message Date
Michael Goin
c9d33c60dc
[UX] Add FlashInfer as default CUDA dependency (#26443) 
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
 2025-10-09 14:10:02 -07:00
elvischenv
44f633dba1
[Flashinfer][gpt-oss] Support FP8-qkv Flashinfer TRTLLM Sinks Attention (#25674) 
Signed-off-by: elvischenv <219235043+elvischenv@users.noreply.github.com>
 2025-10-09 16:13:39 -04:00
roikoren755
4069db3f2e
[Bugfix] Enable padded FP4 quantization (#25947) 
Signed-off-by: Roi Koren <roik@nvidia.com>
 2025-10-09 10:59:41 -07:00
Cyrus Leung
dc7976dd9f
[Misc] Upgrade more code to Python 3.10 (#26463) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-10-09 10:43:53 +01:00
Wentao Ye
f8607863d8
[Feature] Enable E8M0 by Default on Hopper for DeepGEMM, 5% E2E throughput improvement (#26197) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-10-08 15:33:56 +08:00
Benjamin Chislett
caf8b1c084
[Bugfix] Fix MTP+FlashInfer crash when trtllm kernels are available but disabled (#26361) 
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-10-07 22:12:26 +00:00
Cyrus Leung
c0a7b89d8e
[Misc] Move LRUCache into its own file (#26342) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-10-07 15:08:40 +00:00
fxmarty-amd
41f1cf38f2
[Feature][OCP MX] Support mxfp6 and mixed mxfp6-mxfp4 (#21166) 2025-10-07 09:35:26 -04:00
Harry Mellor
b893d661b1
Fix per file ruff ignores related to simplification (#26259) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-10-05 20:31:53 +00:00
Jason Li
6b6e98775f
[NVIDIA] flashinfer TRTLLM attention prefill token limit (#25998) 
Signed-off-by: jasonlizhengjian <jason.li@centml.ai>
Signed-off-by: jasonlizhengjian <jasonlizhengjian@gmail.com>
 2025-10-05 14:24:37 -06:00
Harry Mellor
1c0c68202c
Fix per file ruff ignores related to typing (#26254) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-10-05 16:37:55 +00:00
Harry Mellor
d6953beb91
Convert formatting to use ruff instead of yapf + isort (#26247) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-10-05 07:06:22 -07:00
Cyrus Leung
59a85c366e
[Model] Use merge_by_field_config for MM models (H-L) (#26230) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-10-05 11:54:17 +08:00
Cyrus Leung
44ea85137a
[Model] Support nested structures for TensorSchema (#26212) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-10-04 01:20:32 -07:00
Jun Jiang
4f8f47e87e
Fix undefined symbol: cutlass_moe_mm_sm100 (#26098) 
Signed-off-by: Jun Jiang <jasl9187@hotmail.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
 2025-10-03 15:48:32 +00:00
Wentao Ye
36ce76c632
[Log] Optimize DeepGEMM Missing Log (#26106) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2025-10-02 20:02:26 -06:00
ElizaWszola
502640c3f9
[Perf] Fix and reapply move apply w8a8 block fp8 linear to class (#25696) 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
Signed-off-by: ElizaWszola <elizaw.9289@gmail.com>
Signed-off-by: Luka Govedič <lgovedic@redhat.com>
Signed-off-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Luka Govedič <lgovedic@redhat.com>
 2025-10-02 19:35:13 +00:00
Cyrus Leung
d00d652998
[CI/Build] Replace vllm.entrypoints.openai.api_server entrypoint with vllm serve command (#25967) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-10-02 10:04:57 -07:00
Pavani Majety
ef283548f7
[Bugfix] Fix accuracy issue of TRTLLM FP8 MOE and improve logging (#25895) 
Signed-off-by: Pavani Majety <pmajety@nvidia.com>
 2025-09-30 10:51:31 -04:00
Yongye Zhu
fa7e254a7f
[New Model] DeepSeek-V3.2 (Rebased to Main) (#25896) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: NickLucche <nlucches@redhat.com>
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Signed-off-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com>
Signed-off-by: Lucia Fang <fanglu@meta.com>
Co-authored-by: Chen Zhang <zhangch99@outlook.com>
Co-authored-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Co-authored-by: yewentao256 <zhyanwentao@126.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Lucia Fang <116399278+luccafong@users.noreply.github.com>
Co-authored-by: Lucia Fang <fanglu@meta.com>
Co-authored-by: NickLucche <nlucches@redhat.com>
Co-authored-by: Siyuan Fu <siyuanf@nvidia.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Xiaozhu Meng <mxz297@gmail.com>
Co-authored-by: Barry Kang <43644113+Barry-Delaney@users.noreply.github.com>
 2025-09-30 17:14:41 +08:00
Jialin Ouyang
c216119d64
[Core] GC Debug callback (#24829) 
Signed-off-by: Jialin Ouyang <jialino@meta.com>
Signed-off-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>
Co-authored-by: Jialin Ouyang <jialino@meta.com>
 2025-09-27 17:53:31 +00:00
Jiangyun Zhu
56aafa8c0b
[Misc] fix unique_filepath (#25732) 
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
 2025-09-26 16:56:15 +00:00
Cyrus Leung
12c1287d64
[mypy] Further improve MM type annotations (#25654) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-09-25 10:57:36 +00:00
Tyler Michael Smith
1260180c67
Revert "[Performance] Move apply_w8a8_block_fp8_linear to an op class… (#25607) 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
 2025-09-25 08:05:21 +00:00
Harry Mellor
e7f27ea648
Improve --help for enhanced user experience (#24903) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-09-24 23:08:18 +00:00
Wentao Ye
1f29141258
[Refactor] Use DeepGEMM Col Major TMA Aligned Tensor (#25517) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2025-09-24 18:52:36 -04:00
Shu Wang
54e42b72db
Support mnnvl all2allv from Flashinfer (#21003) 
Signed-off-by: Shu Wang <shuw@nvidia.com>
Signed-off-by: Shu Wang. <shuw@nvidia.com>
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
 2025-09-24 14:38:16 -04:00
Cyrus Leung
9313be5017
[Misc] Improve type annotations for jsontree (#25577) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-09-24 22:49:58 +08:00
Benjamin Chislett
c30b405b8f
[Spec Decode] Enable FlashInfer Spec Decoding (#25196) 
Signed-off-by: Benjamin Chislett <benjamin.chislett@centml.ai>
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
Co-authored-by: lhsjohn <huashuoli@tencent.com>
 2025-09-23 22:29:58 -04:00
Michael Goin
7361ab379f
Remove redundant mutates_args and dispatch_key for direct_register_custom_op (#25512) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-09-23 22:48:40 +00:00
ElizaWszola
63400259d0
[Performance] Move apply_w8a8_block_fp8_linear to an op class (#24666) 
Signed-off-by: ElizaWszola <ewszola@redhat.com>
Signed-off-by: ElizaWszola <elizaw.9289@gmail.com>
Signed-off-by: Luka Govedič <lgovedic@redhat.com>
Signed-off-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Luka Govedič <lgovedic@redhat.com>
 2025-09-23 12:03:10 -07:00
Amir Samani
8c1c81a3de
[core] add nccl symmetric memory for all reduce (#24532) 
Signed-off-by: Amir Samani <asamani@nvidia.com>
Signed-off-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
 2025-09-23 14:33:06 -04:00
Wentao Ye
9949aa2ef1
[Perf] Apply torch.compile for per_block_cast_to_fp8 (#24611) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2025-09-22 19:42:45 -06:00
Luka Govedič
d5e0fca264
[torch.compile] Cleanup compilation tests and custom passes, add debug utils, fix DCE bug (#23091), fix test (#24376), and prep for custom op matching (#24604) (#24542) 
Signed-off-by: Luka Govedič <lgovedic@redhat.com>
Signed-off-by: luka <lgovedic@redhat.com>
Signed-off-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
 2025-09-22 12:30:05 -07:00
Harry Mellor
3d2c56b7a9
Make mypy behave like a proper pre-commit hook (#25313) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-09-22 12:23:45 +00:00
Cyrus Leung
417a164af6
[Misc] Remove unused encoder-decoder error strings (#25374) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-09-22 11:04:32 +00:00
Woosuk Kwon
bc6e542d9f
Remove V0 attention backends (#25351) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-09-21 16:03:28 -07:00
Wenlong Wang
032d661d27
[Docs] Fix warnings in mkdocs build (continued) (#25042) 
Signed-off-by: wwl2755 <wangwenlong2755@gmail.com>
 2025-09-20 11:45:18 +00:00
Nick Hill
b7f186bbb3
[BugFix] Exclude self when checking for port collision (#25286) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-09-20 12:28:31 +08:00
Andrew Sansom
9a4600e4dc
[CORE] Prompt Embeddings Support for v1 Engine (#24278) 
Signed-off-by: Andrew Sansom <andrew@protopia.ai>
Signed-off-by: Andrew Sansom <qthequartermasterman@gmail.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2025-09-19 08:03:09 +08:00
elvischenv
e67a79db03
[Bugfix] Refactor Flashinfer TRTLLM attention kernel selection logic (#24600) 
Signed-off-by: elvischenv <219235043+elvischenv@users.noreply.github.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
 2025-09-17 15:36:29 -07:00
Wentao Ye
de2cc3d867
[Deprecation] Remove DeepGEMM Old Symbol Wrapper (#24902) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2025-09-15 20:03:29 -06:00
Lukas Geiger
1da0f1441d
[Core][Multimodal] Cache supports_kw (#24773) 
Signed-off-by: Lukas Geiger <lukas.geiger94@gmail.com>
 2025-09-13 07:27:04 +00:00
dongluw
a5b84f1cbf
[Core] Shared memory based object store for Multimodal data caching and IPC (#20452) 
Signed-off-by: donglu <donglu@cohere.com>
 2025-09-12 07:54:17 -07:00
Xiaozhu Meng
e42af78b18
[flashinfer] [kernel] support for fp8 kv cache for trtllm prefill attention (#24197) 
Signed-off-by: Xiaozhu <mxz297@gmail.com>
 2025-09-11 14:20:09 -07:00
Boyuan Feng
94e6b2d55f
Allow users to specify kv cache memory size (#21489) 
Signed-off-by: Boyuan Feng <boyuan@meta.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-09-11 13:41:07 +00:00
Charlie Fu
73e688cb79
[ROCm][Feature] Enable Pipeline Parallelism with Ray Compiled Graph on ROCm (#24275) 
Signed-off-by: charlifu <charlifu@amd.com>
 2025-09-09 23:27:35 +00:00
22quinn
0cdd213641
[Misc] Improve Worker process title and logging prefix (#22205) 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
 2025-09-08 21:43:48 -07:00
Zebing Lin
82dfb12e52
[Core] Use sha256 bytes instead of BlockHash to reduce GC overhead (#23673) 
Signed-off-by: linzebing <linzebing1995@gmail.com>
 2025-09-08 21:34:37 -07:00
Nick Hill
752d2e1c36
[Minor] Fix some random typos in comments (#24009) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-08-31 16:42:17 -07:00