xinyun/vllm
1
0
Fork 0
mirror of https://git.datalinker.icu/vllm-project/vllm.git synced 2025-12-27 10:21:48 +08:00
Code Issues Packages Projects Releases Wiki Activity
5,240 Commits 155 Branches 89 Tags
Commit Graph

91 Commits

Author SHA1 Message Date
Aaron Pham
c0efdd655b
[Fix][Structured Output] using vocab_size to construct matcher (#14868) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Signed-off-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
Co-authored-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>
Co-authored-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
 2025-03-17 11:42:45 -04:00
vllmellm
2bb0e1a799
[Bugfix][ROCm] running new process using spawn method for rocm in tests. (#14810) 
Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com>
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
Co-authored-by: TJian <tunjian.tan@embeddedllm.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2025-03-17 11:33:35 +00:00
Lily Liu
8d6cf89526
[V1] [Spec Decode] Support random sampling for spec decode (#13933) 
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-03-16 22:00:20 -07:00
Sibi
a73e183e36
[Misc] Replace os environ to monkeypatch in test suite (#14516) 
Signed-off-by: sibi <85477603+t-sibiraj@users.noreply.github.com>
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: Aaron Pham <contact@aarnphm.xyz>
 2025-03-16 20:35:57 -07:00
Robert Shaw
aecc780dba
[V1] Enable Entrypoints Tests (#14903) 2025-03-16 17:56:16 -07:00
Nick Hill
fc1f67715d
[BugFix][V1] Fix overhead related to bad_words sampling when not in use (#14894) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-03-16 14:53:34 -07:00
Lily Liu
d1ad2a57af
[V1] [Spec Decode] Fix ngram tests (#14878) 2025-03-16 00:29:22 -07:00
Robert Shaw
d4d93db2c5
[V1] V1 Enablement Oracle (#13726) 
Signed-off-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>
Co-authored-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
Co-authored-by: Michael Goin <michael@neuralmagic.com>
 2025-03-14 22:02:20 -07:00
Russell Bryant
46f98893dd
[V1] Fix model parameterization for structured output tests (#14833) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-03-14 20:55:18 +00:00
afeldman-nm
02fcaa3d0a
[V1] Detokenizer: Respect Stop Tokens + not include_stop_str_in_output (#14624) 
Signed-off-by: Andrew Feldman <afeldman@neuralmagic.com>
 2025-03-13 19:07:34 +00:00
Nick Hill
f5d3acd474
[BugFix][V1] Fix parallel sampling finishing/aborts (#14512) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-03-12 10:29:48 -07:00
Benjamin Chislett
5c538c37b2
[V1][Bugfix][Spec Decode] Fix incorrect outputs in V1 speculative decoding due to batch indexing (#14645) 
Signed-off-by: Benjamin Chislett <benjamin.chislett@centml.ai>
 2025-03-11 22:12:41 -07:00
Aaron Pham
77a318bd01
[V1][Core] Support MistralTokenizer for Structured Output (#14625) 
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
 2025-03-12 10:40:09 +08:00
Russell Bryant
4bf82d4b90
[V1] Add regex structured output support with xgrammar (#14590) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-03-11 23:03:44 +08:00
22quinn
eb8b5eb183
[V1] Support bad_words in sampler (#13376) 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-03-08 14:50:26 -08:00
Alexander Matveev
cb8bdfade2
[V1] TPU - Add tensor parallel support via Ray (#13618) 
Signed-off-by: Alexander Matveev <amatveev@redhat.com>
 2025-03-08 08:19:38 -05:00
afeldman-nm
ef64044079
[V1] Prompt logprobs + APC compatibility; prompt logprobs reqs cannot fill APC (#13949) 2025-03-08 01:48:12 +00:00
Nick Hill
8ed5421aaa
[V1] Eagerly remove finished requests from the batch (#14388) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-03-07 10:56:00 -08:00
Aaron Pham
80e9afb5bc
[V1][Core] Support for Structured Outputs (#12388) 
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-03-07 07:19:11 -08:00
Himanshu Jaju
cd579352bf
[V1] Do not detokenize if sampling param detokenize is False (#14224) 
Signed-off-by: Himanshu Jaju <hj@mistral.ai>
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-03-06 10:40:24 -08:00
Harry Mellor
bf0560bda9
Reinstate best_of for V0 (#14356) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-03-06 08:34:22 -08:00
Lucas Wilkinson
f6bb18fd9a
[BugFix] MLA + V1, illegal memory access and accuracy issues (#14253) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
 2025-03-05 17:10:13 -08:00
Lu Fang
53ea6ad830
[V1][Easy] Add empty allowed_token_ids in the v1 sampler test (#14308) 
Signed-off-by: Lu Fang <lufang@fb.com>
 2025-03-05 21:41:18 +00:00
Vincent
a4f1ee35d6
Deprecate best_of Sampling Parameter in anticipation for vLLM V1 (#13997) 
Signed-off-by: vincent-4 <vincentzhongy+githubvincent4@gmail.com>
Signed-off-by: Brayden Zhong <b8zhong@uwaterloo.ca>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Brayden Zhong <b8zhong@uwaterloo.ca>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-03-05 20:22:43 +00:00
Robert Shaw
257e200a25
[V1][Frontend] Add Testing For V1 Runtime Parameters (#14159) 
Signed-off-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>
 2025-03-05 14:18:55 +00:00
Nick Hill
5db6b2c961
[V1][BugFix] Fix remaining sync engine client shutdown errors/hangs (#13869) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-03-04 15:06:47 +00:00
Harry Mellor
cf069aa8aa
Update deprecated Python 3.8 typing (#13971) 2025-03-02 17:34:51 -08:00
Chen Zhang
28943d36ce
[v1] Move block pool operations to a separate class (#13973) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>
 2025-02-28 20:53:31 +00:00
Chen Zhang
e7bd944e08
[v1] Cleanup the BlockTable in InputBatch (#13977) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-02-28 19:03:16 +00:00
Lily Liu
5629f26df7
[V1][Spec Decode] Change Spec Decode Rejection Sampling API (#13729) 2025-02-25 18:14:48 -08:00
afeldman-nm
befc402d34
[V1] V1 engine implements parallel sampling (AsyncLLM and LLMEngine) (#10980) 
Signed-off-by: Andrew Feldman <afeldman@neuralmagic.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-02-24 08:29:41 -08:00
Nick Hill
cbae7af552
[V1][BugFix] Fix engine core client shutdown hangs (#13298) 
Even though ZMQ context.destroy() is meant to close open sockets before terminating the context, it appears to be necessary to do this explicitly or else it can hang in the context.term() method.

Close zmq sockets explicitly before terminating context, make shutdown of client resource more robust, shut down engine core process prior to terminating zmq context.

Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-02-23 13:07:43 -08:00
youkaichao
eb24dc4a45
[v1] torchrun compatibility (#13642) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2025-02-23 22:47:24 +08:00
Lu Fang
bb78fb318e
[v1] Support allowed_token_ids in v1 Sampler (#13210) 
Signed-off-by: Lu Fang <lufang@fb.com>
 2025-02-22 14:13:05 +08:00
Nick Hill
caf7ff4456
[V1][Core] Generic mechanism for handling engine utility (#13060) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-02-19 17:09:22 +08:00
Nick Hill
30172b4947
[V1] Optimize handling of sampling metadata and req_ids list (#13244) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-02-18 12:15:33 -08:00
Murali Andoorveedu
a4d577b379
[V1][Tests] Adding additional testing for multimodal models to V1 (#13308) 
Signed-off-by: andoorve <37849411+andoorve@users.noreply.github.com>
 2025-02-18 09:53:14 -08:00
Woosuk Kwon
cd4a72a28d
[V1][Spec decode] Move drafter to model runner (#13363) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-02-17 15:40:12 -08:00
Woosuk Kwon
4c21ce9eba
[V1] Get input tokens from scheduler (#13339) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-02-17 11:01:07 -08:00
Lily Liu
80f63a3966
[V1][Spec Decode] Ngram Spec Decode (#12193) 
Signed-off-by: LiuXiaoxuanPKU <lilyliupku@gmail.com>
 2025-02-15 18:05:11 -08:00
Cody Yu
9206b3d7ec
[V1][PP] Run engine busy loop with batch queue (#13064) 2025-02-15 03:59:01 -08:00
Woosuk Kwon
e7eea5a520
[V1][CI] Fix failed v1-test because of min_p (#13316) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-02-14 17:29:51 -08:00
Aoyu
a12934d3ec
[V1][Core] min_p sampling support (#13191) 
Signed-off-by: Aoyu <aoyuzhan@amazon.com>
Co-authored-by: Aoyu <aoyuzhan@amazon.com>
 2025-02-14 15:50:05 -08:00
Lu Fang
6224a9f620
Support logit_bias in v1 Sampler (#13079) 2025-02-14 04:34:59 -08:00
Kero Liang
b0ccfc565a
[Bugfix][V1] GPUModelRunner._update_states should return True when there is a finished request in batch (#13126) 2025-02-13 22:39:20 -08:00
Harry Mellor
f2b20fe491
Consolidate Llama model usage in tests (#13094) 2025-02-13 22:18:03 -08:00
Mark McLoughlin
75e6e14516
[V1][Metrics] Add several request timing histograms (#12644) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2025-02-11 10:14:00 -05:00
Cody Yu
41c5dd45b9
[V1][Metrics] Add GPU prefix cache hit rate % gauge (#12592) 2025-02-11 08:27:25 +00:00
Woosuk Kwon
3243158336
[V1] Move KV block hashes from Request to KVCacheManager (#12922) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-02-07 19:14:10 -08:00
afeldman-nm
0630d4537a
[V1] Logprobs and prompt logprobs support (#9880) 
This PR is adding support for sample logprobs & prompt logprobs to vLLM v1.

New behavior:

- During model execution, model runner computes sample logprobs (if user-provided logprobs setting is not None) and prompt logprobs (if user-provided prompt_logprobs setting is not None). For both sample and prompt logprobs, the engine core returns 3 vectors: token ids, token logprob values, token ranks. Ranks reflect tokens' 1-indexed positions in the vocabulary vector after sorting the vocabulary by log probability in descending order.
- In scheduler.update_from_output(), sample and prompt logprobs are incorporated into the EngineCoreOutput data structure which is transferred to the engine client. If multiprocessing is enabled, then sample and prompt logprobs will be (de)serialized when the EngineCoreOutput data structure is (de)serialized.
- During output processing, the LogprobsProcessor transforms the triplet of token ids, token logprobs values, and token ranks into the OpenAI-compatible List[Dict[token id,Logprob]] format (for sample and prompt logprobs respectively.)
- Each Logprob instance (whether sample- or prompt-) consists of a token's log-probability, rank, and detokenized string representation. Note that logprob detokenization is handled by the LogprobsProcessor not the detokenizer.

Signed-off-by: Andrew Feldman <afeldman@neuralmagic.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Signed-off-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>


Co-authored-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-02-07 07:26:20 -08:00