xinyun/vllm
1
0
Fork 0
mirror of https://git.datalinker.icu/vllm-project/vllm.git synced 2025-12-09 23:15:31 +08:00
Code Issues Packages Projects Releases Wiki Activity
6,073 Commits 155 Branches 89 Tags
Commit Graph

179 Commits

Author SHA1 Message Date
Nick Hill
b07bf83c7d
[BugFix] Avoid race conditions in zero-copy tensor transmission (#17203) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-04-26 06:00:07 +00:00
Zijing Liu
53e8cf53a4
[V1][Metrics] Allow V1 AsyncLLM to use custom logger (#14661) 
Signed-off-by: Zijing Liu <liuzijing2014@gmail.com>
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-04-25 22:05:40 -07:00
Benjamin Chislett
a0e619e62a
[V1][Spec Decode] EAGLE-3 Support (#16937) 
Signed-off-by: Bryan Lu <yuzhelu@amazon.com>
Signed-off-by: Benjamin Chislett <benjamin.chislett@centml.ai>
Co-authored-by: Bryan Lu <yuzhelu@amazon.com>
 2025-04-25 15:43:07 -07:00
Sangyeon Cho
6aae216b4e
[Bugfix] remove fallback in guided_json (int range, patterns) (#16725) 
Signed-off-by: csy1204 <josang1204@gmail.com>
Co-authored-by: 조상연[플레이스 AI] <sang-yeon.cho@navercorp.com>
 2025-04-25 06:54:43 +00:00
Mark McLoughlin
340d7b1b21
[V1][Spec Decoding] Add num_drafts and num_accepted_tokens_per_position metrics (#16665) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2025-04-24 08:57:40 -07:00
Rui Qiao
c0dfd97519
[V1][PP] Optimization: continue scheduling prefill chunks (#17080) 
Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
 2025-04-24 05:27:08 -07:00
Harry Mellor
0a05ed57e6
Simplify TokenizerGroup (#16790) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-04-24 04:43:56 -07:00
Michael Goin
14288d1332
Disable enforce_eager for V1 TPU sampler and structured output tests (#17016) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-04-24 02:50:09 -07:00
Travis Johnson
3cde34a4a4
[Frontend] Support guidance:no-additional-properties for compatibility with xgrammar (#15949) 
Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
 2025-04-23 18:34:41 +00:00
Nick Hill
1e013fa388
[V1][DP] More robust DP/EP dummy request coordination (#16277) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-04-22 19:12:15 -07:00
Chenyaaang
83d933718c
[Core][V1][TPU] Enable structured decoding on TPU V1 (#16499) 
Signed-off-by: Chenyaaang <chenyangli@google.com>
 2025-04-22 18:05:23 -06:00
Woosuk Kwon
c4ab9f3e71
[V1] Remove pre-allocation for KV cache (#16941) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-04-22 00:52:18 -07:00
Chauncey
acba33a0f1
[Bugfix] Fix the issue where llm.generate cannot be called repeatedly after setting GuidedDecodingParams (#16767) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Russell Bryant <rbryant@redhat.com>
 2025-04-22 06:02:20 +00:00
Jeffrey Li
0e4254492f
[Bugfix]: fix issue with n>1 sampling on v1 requests overriding each other (#16863) 
Signed-off-by: Jeffrey Li <jeffrey.dot.li@gmail.com>
 2025-04-22 11:40:19 +08:00
Nicolò Lucchesi
fa3bba2a53
[TPU][V1] Enable Top-P (#16843) 
Signed-off-by: NickLucche <nlucches@redhat.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
 2025-04-22 00:46:07 +00:00
Michael Goin
986537f1c3
[V1] V1 FlashInfer Attention (#16684) 
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Aurick Qiao <qiao@aurick.net>
 2025-04-22 00:38:41 +00:00
Nicolò Lucchesi
210207525e
[TPU][V1] Capture multimodal encoder during model compilation (#15051) 
Signed-off-by: Michael Goin <mgoin64@gmail.com>
Signed-off-by: NickLucche <nlucches@redhat.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Siyuan Liu <lsiyuan@google.com>
 2025-04-21 18:36:59 -06:00
Chengji Yao
471fe65630
[TPU][V1] Implicitly adjust page size when there's SMEM OOM (#16871) 
Signed-off-by: Chengji Yao <chengjiyao@google.com>
 2025-04-21 15:43:13 -06:00
Woosuk Kwon
3a0fba5cf4
[V1][Spec Decode] Handle draft tokens beyond max_model_len (#16087) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-04-21 12:38:50 -07:00
qizixi
bb3605db85
[Bugfix] Fix v1/spec_decode/test_ngram.py (#16895) 
Signed-off-by: qizixi <qizixi@meta.com>
 2025-04-20 20:54:29 -07:00
Staszek Paśko
87aaadef73
Serialize tensors using int8 views (#16866) 
Signed-off-by: Staszek Pasko <staszek@gmail.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-04-19 10:28:34 -07:00
vie-serendipity
d9737ca1c6
[V1][Misc] stop update prefix cache stats when logs_stats is disabled (#16460) 
Signed-off-by: vie-serendipity <2733147505@qq.com>
 2025-04-19 02:25:19 -07:00
Yihua Cheng
3408e47159
[P/D][V1] KV Connector API V1 (#15960) 
Signed-off-by: ApostaC <yihua98@uchicago.edu>
Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>
Signed-off-by: remi <remi@mistral.ai>
Co-authored-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Rémi Delacourt <54138269+Flechman@users.noreply.github.com>
Co-authored-by: Tyler Michael Smith <tysmith@redhat.com>
 2025-04-17 13:22:40 -07:00
Nicolò Lucchesi
eb5819b2d9
[V1][TPU] Enable Top K (#15489) 
Signed-off-by: NickLucche <nlucches@redhat.com>
Signed-off-by: Hyesoo Yang <hyeygit@gmail.com>
Co-authored-by: Hyesoo Yang <hyeygit@gmail.com>
 2025-04-17 18:18:11 +00:00
Nicolò Lucchesi
5989f4684d
[TPU][V1] Fix padding recompilation when max-num-batched-tokens is not even (#16726) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2025-04-17 18:09:57 +00:00
Robert Shaw
2b05b8ce69
[V1][Frontend] Improve Shutdown And Logs (#11737) 
Signed-off-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>
Signed-off-by: Andrew Feldman <afeldman@neuralmagic.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Andrew Feldman <afeldman@neuralmagic.com>
Co-authored-by: afeldman-nm <156691304+afeldman-nm@users.noreply.github.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-04-16 19:48:34 -07:00
Staszek Paśko
3092375e27
[V1][Performance] Implement custom serializaton for MultiModalKwargs [Rebased] (#16432) 
Signed-off-by: Staszek Pasko <staszek@gmail.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-04-16 19:28:32 -07:00
Shanshan Shen
976711d9db
[V1][Structured Output] Move xgrammar related utils to backend_xgrammar.py (#16578) 
Signed-off-by: shen-shanshan <467638484@qq.com>
 2025-04-16 17:01:36 +08:00
Nicolò Lucchesi
b3f2fddd17
[TPU][V1] Fix exponential padding when max-num-batched-tokens is not a power of 2 (#16596) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2025-04-14 17:01:05 +00:00
Lily Liu
f49e5aff11
[V1][Spec Decode] KV cache slots for eagle heads (#16370) 
Signed-off-by: LiuXiaoxuanPKU <lilyliupku@gmail.com>
 2025-04-12 19:42:51 -07:00
leon-seidel
e92d7085bf
[Feature][V1] Add xgrammar to support minLength, maxLength with test (#16516) 
Signed-off-by: Leon Seidel <leon.seidel@fau.de>
 2025-04-11 23:22:07 -07:00
Nick Hill
41cc883c29
[BugFix] Handle non-contiguous tensors properly when serializing (#16492) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-04-11 17:54:06 -07:00
Michael Goin
aa3b3d76e0
Enforce valid max_num_batched_tokens when disable_chunked_mm_input=True (#16447) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-04-11 08:09:52 +00:00
Nicolò Lucchesi
3cc9af88ff
[TPU][V1] Disable per-request seed/Generator (#16172) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2025-04-10 17:05:44 -04:00
Nick Hill
dd143ef541
[V1] Zero-copy tensor/ndarray serialization/transmission (#13790) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-04-10 19:23:14 +00:00
Lily Liu
e8224f3dca
[V1][Spec Decode] Eagle Model loading (#16035) 
Signed-off-by: LiuXiaoxuanPKU <lilyliupku@gmail.com>
 2025-04-10 11:21:48 -07:00
Chengji Yao
a454748544
[TPU][V1] Refine tpu_model_runner to mitigate future recompilation issues (#16275) 
Signed-off-by: Chengji Yao <chengjiyao@google.com>
 2025-04-09 18:51:51 -06:00
Chengji Yao
b1eb4ca152
[TPU] Update PyTorch/XLA (#16288) 
Signed-off-by: Chengji Yao <chengjiyao@google.com>
 2025-04-09 14:46:32 +08:00
rongfu.leng
4716377fbc
[Feature] Estimate max-model-len use available KV cache memory (#16168) 
Signed-off-by: rongfu.leng <rongfu.leng@daocloud.io>
 2025-04-08 19:12:51 -07:00
Michael Goin
8e5314a468
[V1] Add disable_chunked_mm_input arg to disable partial mm input prefill (#15837) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-04-07 23:24:07 -07:00
Roger Wang
f2ebb6f541
[V1] Scatter and gather placeholders in the model runner (#16076) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Roger Wang <ywang@roblox.com>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Jennifer Zhao <ai.jenniferzhao@gmail.com>
 2025-04-08 10:43:41 +08:00
Nick Hill
7f6d47c1a2
[V1][BugFix] Exit properly if engine core fails during startup (#16137) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-04-07 15:30:15 -07:00
Cyrus Leung
66d433b94f
[V1] Revert the default max_num_seqs to V0 values for most hardware (#16158) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-04-07 13:54:36 -04:00
Roger Wang
af51d80fa1
Revert "[V1] Scatter and gather placeholders in the model runner" (#16075) 2025-04-04 14:50:57 -07:00
Cyrus Leung
f5722a5052
[V1] Scatter and gather placeholders in the model runner (#15712) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Roger Wang <ywang@roblox.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Roger Wang <ywang@roblox.com>
 2025-04-04 21:26:44 +00:00
Mark McLoughlin
a35a8a8392
[V1][Spec Decode] Avoid logging useless nan metrics (#16023) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2025-04-04 08:52:41 -07:00
iefgnoix
b6be6f8d1e
[TPU] Support sliding window and logit soft capping in the paged attention kernel for TPU. (#15732) 
Signed-off-by: Xiongfei Wei <isaacwxf23@gmail.com>
 2025-04-03 14:23:28 -07:00
Hyesoo Yang
1b84eff03a
[V1][TPU] TPU-optimized top-p implementation (avoids scattering). (#15736) 
Signed-off-by: Hyesoo Yang <hyeygit@gmail.com>
Co-authored-by: root <root@t1v-n-822696b7-w-0.us-central2-b.c.tpu-prod-env-large-adhoc.internal>
 2025-04-02 17:18:08 -07:00
Russell Bryant
14e53ed11f
[V1] Fix json_object support with xgrammar (#15488) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-04-02 02:00:08 -07:00
Mark McLoughlin
a79cc68b3a
[V1][Metrics] Initial speculative decoding metrics (#15151) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2025-04-01 10:45:04 -07:00