9 Commits

Author	SHA1	Message	Date
youkaichao	20cfcdec99	[Core][Optimization] change python dict to pytorch tensor for blocks to swap (#4659)	2024-05-08 12:07:05 -07:00
youkaichao	469f85c782	[Core][Optimization] change copy-on-write from dict[int, list] to list (#4648)	2024-05-07 11:06:32 -07:00
SangBin Cho	0f8a91401c	[Core] Ignore infeasible swap requests. (#4557)	2024-05-02 14:31:20 -07:00
SangBin Cho	cf8cac8c70	[mypy][6/N] Fix all the core subdirectory typing (#4450) ... Co-authored-by: Cade Daniel <edacih@gmail.com>	2024-05-02 03:01:00 +00:00
leiwen83	24750f4cad	[Core] Enable prefix caching with block manager v2 enabled (#4142) ... Co-authored-by: Lei Wen <wenlei03@qiyi.com> Co-authored-by: Sage Moore <sagemoore@utexas.edu>	2024-05-01 11:20:32 -07:00
SangBin Cho	37e84a403d	[Typing] Fix Sequence type GenericAlias only available after Python 3.9. (#4092)	2024-04-15 14:47:31 -07:00
SangBin Cho	09473ee41c	[mypy] Add mypy type annotation part 1 (#4006)	2024-04-12 14:35:50 -07:00
Cade Daniel	93deb0b38f	[Speculative decoding 4/9] Lookahead scheduling for speculative decoding (#3250)	2024-04-01 22:55:24 +00:00
Cade Daniel	14ccd94c89	[Core][Bugfix]Refactor block manager for better testability (#3492)	2024-03-27 23:59:28 -07:00