xinyun/vllm
1
0
Fork 0
mirror of https://git.datalinker.icu/vllm-project/vllm.git synced 2025-12-23 08:05:01 +08:00
Code Issues Packages Projects Releases Wiki Activity
12,170 Commits 155 Branches 89 Tags
Commit Graph

172 Commits

Author SHA1 Message Date
Or Ozeri
4c6fd25880
kv_transfer: Rename the shared storage connectors (#30201) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2025-12-08 20:46:09 -08:00
Cyrus Leung
e83b7e379c
Revert "[Renderer] Separate out RendererConfig from ModelConfig (#30145)" (#30199) 2025-12-07 00:00:22 -08:00
Cyrus Leung
27f4c2fd46
[Renderer] Separate out RendererConfig from ModelConfig (#30145) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-12-06 23:15:42 -08:00
rasmith
feecba09af
[CI/Build][AMD] Use float16 in test_reset_prefix_cache_e2e to avoid accuracy issues (#29997) 
Signed-off-by: Randall Smith <ransmith@amd.com>
Co-authored-by: Randall Smith <ransmith@amd.com>
 2025-12-05 08:42:25 +00:00
Mark McLoughlin
899e2ef558
[Core] Fix standalone runs of test_reset_prefix_cache_e2e (#29899) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2025-12-04 16:22:03 +08:00
Chauncey
0a9caca9f5
[Bugfix] fix --scheduling-policy=priority & n>1 crashes engine (#29764) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-12-02 22:42:28 +00:00
Harry Mellor
951445a52d
Remove default values from InitVars so that they're not stored (#29859) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-12-02 12:16:37 +00:00
Zhuohan Li
d0cd728907
[Core] Support reseting all running requests' KV while calling reset_prefix_cache (#28827) 
Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-12-02 02:25:05 +00:00
shivampr
cabc77cc86
[Core][Observability] Add KV cache residency metrics (#27793) 
Introduces three new Prometheus histograms for fine-grained observability of KV cache residency behavior:

vllm:kv_block_lifetime_seconds — total lifetime from allocation to free
vllm:kv_block_idle_before_evict_seconds — idle duration before eviction
vllm:kv_block_reuse_gap_seconds — time between consecutive reuses of the same block

These metrics help operators analyze KV cache efficiency, reuse patterns, and eviction timing beyond simple utilization rates.

Implementation uses monotonic timestamps for accuracy, 1% sampling for minimal overhead (~48 bytes/block), and is fully thread-safe with zero runtime cost when disabled.

Two new runtime flags are introduced:

--kv-cache-metrics – enable KV cache residency metrics
--kv-cache-metrics-sample – control sampling ratio (default: 0.01)

Signed-off-by: Shivam <shivamprasad91@gmail.com>
 2025-12-01 18:27:53 +00:00
Marcin Ostrowski
5cfa967efa
[Bugfix] TypeError: 'NoneType' object is not callable (#29414) 
Signed-off-by: Marcin Ostrowski <marcinx.ostrowski@intel.com>
 2025-12-01 13:16:44 +00:00
maang-h
c7ba1f6bc7
[BugFix] Fix ValueError in NewRequestData repr methods (#29392) 
Signed-off-by: maang <maang_h@163.com>
 2025-11-28 13:42:30 +08:00
Yifan Qiao
48ddb02b79
[Hybrid Allocator] Support KV cache groups with different block_size (#29143) 
Signed-off-by: Yifan Qiao <yifanqiao@berkeley.edu>
Co-authored-by: Chen Zhang <zhangch99@outlook.com>
 2025-11-25 10:30:57 -05:00
wang.yuqi
67fc16cd8c
[Bugfix] If chunked_prefill is disabled, end the scheduling early. (#28911) 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
 2025-11-25 16:06:09 +08:00
Chen Zhang
71df2a57ef
[Hybrid Allocator] Better layer padding strategy for gpt-oss eagle (#29303) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-11-24 14:28:32 -08:00
Cyrus Leung
5a4802588e
[Misc] Further clean up chunked prefill and prefix caching init (#29186) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-11-22 19:34:15 +08:00
Mark McLoughlin
c6fa3895e9
[KV Connector] Fix async connector prefix cache metrics (#28585) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
 2025-11-21 17:45:00 -05:00
Jialin Ouyang
30b9c67743
Revert "[Redo] #26368 (#28771)" (#29121) 
Signed-off-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>
 2025-11-20 21:27:45 -08:00
Cyrus Leung
98b4d389ed
[Redo] #26368 (#28771) 
Signed-off-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>
 2025-11-14 22:47:41 -08:00
Nick Hill
ac86bff8cb
Revert "[Core] Performance: Use list[np.ndarray] instead of list[list… (#28773) 2025-11-14 20:24:00 -08:00
Jialin Ouyang
186352b270
[Core] Performance: Use list[np.ndarray] instead of list[list[int]] for output tokens for GC optimization (#26368) 
Signed-off-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>
 2025-11-14 16:04:04 -08:00
Marcin Ostrowski
0de4f217ab
[Bugfix] TypeError: 'NoneType' object is not callable (#27410) 
Signed-off-by: Marcin Ostrowski <marcinx.ostrowski@intel.com>
 2025-11-14 21:13:53 +00:00
Cyrus Leung
e2741f6cbc
[Chore] Rename SchedulerConfig.chunked_prefill_enabled (#28735) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-11-14 18:39:57 +00:00
Cyrus Leung
511a6b611d
[Config] Clean up SchedulerConfig initialization (#28665) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-11-14 22:41:02 +08:00
Mark McLoughlin
6e25b1cddf
[KV Connector] Test async mode in scheduler tests (#28550) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2025-11-13 18:30:59 -05:00
Andy Lo
58ce8d12b7
[BugFix] Priority scheduling and spec tokens preemption (#28558) 
Signed-off-by: Andy Lo <andy@mistral.ai>
 2025-11-12 20:29:21 +00:00
Chenguang Zheng
4ccffe561f
[Core] Encoder separation for Encode-Prefill-Decode Disaggregation (#25233) 
Signed-off-by: n00909098 <nguyen.kha.long@huawei.com>
Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com>
Signed-off-by: herotai214 <herotai214@gmail.com>
Signed-off-by: Khuong Le <khuong.le.manh@huawei.com>
Signed-off-by: Khuong Le <lemanhkhuong2611@gmail.com>
Co-authored-by: n00909098 <nguyen.kha.long@huawei.com>
Co-authored-by: knlnguyen1802 <knlnguyen1802@gmail.com>
Co-authored-by: herotai214 <herotai214@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Khuong Le <khuong.le.manh@huawei.com>
Co-authored-by: Khuong Le <lemanhkhuong2611@gmail.com>
 2025-11-11 18:58:33 -08:00
Kuntai Du
86dca07d9b
[Hybrid allocator + kv connector] revert connector test changes related to hybrid allocator (#28011) 
Signed-off-by: KuntaiDu <kuntai@uchicago.edu>
 2025-11-05 10:36:31 +00:00
Nick Hill
938a81692e
[AsyncScheduling] Don't schedule past request max_tokens (#27922) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-11-04 17:06:28 +00:00
Biswa Panda
1bf43ae35d
[BugFix][LoRA] use adapter_id instead of id field of lora_request (#27728) 
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
 2025-11-03 10:08:08 +08:00
Nick Hill
0cdbe7b744
[Core] Async scheduling + structured outputs compatibility (#26866) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-11-01 00:35:04 +00:00
Kuntai Du
b853540388
[Core][Hybrid allocator + kv connector 1/n] Enable hybrid allocator + KV cache connector (#25712) 
Signed-off-by: KuntaiDu <kuntai@uchicago.edu>
Signed-off-by: Kuntai Du <kuntai@uchicago.edu>
 2025-10-24 23:34:18 -07:00
Tova Movshovitz
88afa11010
[Metrics] [KVConnector] Add connector prefix cache hit rate stats (#26245) 
Signed-off-by: tovam <tovam@pliops.com>
 2025-10-23 12:21:08 +02:00
Andrew Sansom
ff93cc8c84
[CORE] Support Prefix Caching with Prompt Embeds (#27219) 
Signed-off-by: Andrew Sansom <andrew@protopia.ai>
 2025-10-22 22:18:07 -07:00
Sage
1651003c35
[Prefix Cache] Use LoRA name for consistent KV-cache block hashing (#27211) 
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
 2025-10-22 18:13:03 +00:00
dongbo910220
8a297115e2
[Chore] Separate out hashing utilities from vllm.utils (#27151) 
Signed-off-by: dongbo910220 <1275604947@qq.com>
 2025-10-19 11:09:38 +08:00
iAmir97
1d165d6d85
[Chore] Separate out vllm.utils.mem_utils (#27143) 
Signed-off-by: iAmir97 <Amir.balwel@embeddedllm.com>
Signed-off-by: iAmir97 <71513472+iAmir97@users.noreply.github.com>
Co-authored-by: iAmir97 <Amir.balwel@embeddedllm.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-10-18 10:06:59 +00:00
Tahsin Tunan
43721bc67f
[CI] Replace large models with tiny alternatives in tests (#24057) 
Signed-off-by: Tahsin Tunan <tahsintunan@gmail.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-10-16 15:51:27 +01:00
Nick Hill
4aed506b65
[Core] Streamline some structured output related code (#26737) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-10-14 23:27:44 +00:00
Harry Mellor
8fcaaf6a16
Update Optional[x] -> x | None and Union[x, y] to x | y (#26633) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-10-12 09:51:31 -07:00
Harry Mellor
7c12763b24
Fix some typing issues found by mypy==1.18.2 (#26596) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-10-10 18:21:25 +00:00
Chen Zhang
606b00e80f
[bugfix][DCP] fix block_size of hash in DCP prefix caching (#26296) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-10-10 03:02:49 -07:00
Cyrus Leung
ad430a67ca
[Metrics] Log multi-modal cache stats and fix reset (#26285) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-10-10 01:45:55 -07:00
Qier Li
d17f0fbf30
[Core][KVConnector] Propagate all tokens on resumed preemptions (#24926) 
Signed-off-by: Qier Li <kevin44036@gmail.com>
Co-authored-by: Qier Li <qier@fb.com>
 2025-10-09 14:43:31 +08:00
Elaine Zhao
f08919b7d1
[Bugfix] Respect min_tokens in scheduler stop check (#26317) 
Signed-off-by: Elaine Zhao <elaineyz@amazon.com>
 2025-10-08 14:08:24 -07:00
Cyrus Leung
1e4ecca1d0
[V0 Deprecation] Remove VLLM_USE_V1 from tests (#26341) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-10-07 15:42:31 +00:00
Grant Holmes (Ren)
d100d78eb3
Optimize KV cache distribution for asymmetric pipeline parallelism (#25164) 
Signed-off-by: gholmes829 <g.holmes429@gmail.com>
 2025-10-07 09:20:30 +00:00
Harry Mellor
4e256cadc2
Remove all references to yapf as it's no longer used (#26251) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-10-05 09:18:11 -07:00
Harry Mellor
d6953beb91
Convert formatting to use ruff instead of yapf + isort (#26247) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-10-05 07:06:22 -07:00
Huamin Li
7d6b03381e
[CI Failure] fix_test_auto_prefix_cache_support (#26053) 
Signed-off-by: Huamin Li <3ericli@gmail.com>
 2025-10-04 02:44:49 -07:00
Reza Barazesh
bc546f76a1
[CI] Move applicable tests to CPU (#24080) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-09-30 14:45:20 +01:00