44 Commits

Author	SHA1	Message	Date
Woosuk Kwon	8917782af6	Add a system logger (#85)	2023-05-08 23:03:35 -07:00
Woosuk Kwon	c9d5b6d4a8	Replace FlashAttention with xformers (#70)	2023-05-05 02:01:08 -07:00
Woosuk Kwon	189ae23133	Use dtype from model config & Add Dolly V2 (#63)	2023-05-04 03:05:37 -07:00
Woosuk Kwon	e070829ae8	Support bfloat16 data type (#54)	2023-05-03 14:09:44 -07:00
Zhuohan Li	27f1410d06	New weight loader without np copy (#52)	2023-05-03 15:32:04 +08:00
Zhuohan Li	4858f3bb45	Add an option to launch cacheflow without ray (#51)	2023-04-30 15:42:17 +08:00
Woosuk Kwon	0f4b32199e	Support various block sizes & Change default block size to 16 (#38)	2023-04-15 09:03:24 -07:00
Woosuk Kwon	84eee24e20	Collect system stats in scheduler & Add scripts for experiments (#30)	2023-04-12 15:03:49 -07:00
Woosuk Kwon	b9926f7f66	Support block size 32 (#35)	2023-04-09 23:07:18 -07:00
Woosuk Kwon	ee88a7e5f3	Add an option to use dummy model weights (#33)	2023-04-08 23:36:12 -07:00
Woosuk Kwon	12659a0bd7	Add CUDA graph-based all reduce launcher (#26)	2023-04-05 11:16:57 -07:00
Woosuk Kwon	7a7929abe8	Implement preemption via recomputation & Refactor scheduling logic (#12)	2023-03-30 14:51:46 -07:00
Woosuk Kwon	80a2f812f1	Implement LLaMA (#9) ... Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>	2023-03-30 12:25:32 +08:00
Woosuk Kwon	64e0e38314	Add cache watermark to avoid frequent cache eviction (#11)	2023-03-29 16:38:48 -07:00
Zhuohan Li	721fa3df15	FastAPI-based working frontend (#10)	2023-03-29 14:48:56 +08:00
Woosuk Kwon	d359cda5fa	Minor	2023-03-26 08:00:39 +00:00
Woosuk Kwon	cfae35b861	Add miscellaneous updates (#8)	2023-03-13 13:48:38 -07:00
Woosuk Kwon	e9d3f2ff77	Add memory analyzer & utomatically configure KV cache size (#6)	2023-03-11 23:23:14 -08:00
Woosuk Kwon	1a7eb7da61	Support beam search & parallel generation (#7)	2023-03-10 09:58:21 -08:00
Woosuk Kwon	0deacbce6e	Implement single_query_cached_kv_attention kernel (#3)	2023-03-01 15:02:19 -08:00
Woosuk Kwon	6aef2278f4	[Minor] Fix printing format	2023-02-24 11:56:06 +00:00
Woosuk Kwon	1132fae0ca	Add Frontend	2023-02-24 11:46:43 +00:00
Woosuk Kwon	ef6098ec51	Merge pre_step and step	2023-02-24 10:36:08 +00:00
Woosuk Kwon	53f70e7334	Reduce the number of states in scheduler	2023-02-24 10:22:39 +00:00
Woosuk Kwon	afdbe5d373	[WIP] Add server script	2023-02-24 01:33:37 +00:00
Woosuk Kwon	fdd0f2f472	Minor	2023-02-23 20:23:47 +00:00
Woosuk Kwon	4f6f4967f6	Add get_block_table	2023-02-23 07:55:14 +00:00
Woosuk Kwon	331fa0b042	Implement scheduler.step & Add a threshold for batch size	2023-02-23 07:54:20 +00:00
Woosuk Kwon	2729087efe	Fix a ref count bug in swapping	2023-02-14 09:20:12 +00:00
Woosuk Kwon	c128c2ed03	Fix double free	2023-02-14 09:14:05 +00:00
Woosuk Kwon	c80ed212a1	Minor	2023-02-14 09:06:27 +00:00
Woosuk Kwon	263e91b386	Minor	2023-02-14 09:05:39 +00:00
Woosuk Kwon	be1e2163c9	Fix memory leak in swapping	2023-02-14 08:46:33 +00:00
Woosuk Kwon	7e5f604e68	Fix bugs in scheduler	2023-02-14 02:25:32 +00:00
Woosuk Kwon	1f739f9b03	Fix a bug in swap_in and swap_out	2023-02-14 02:12:58 +00:00
Woosuk Kwon	b1644f764f	Fix can_swap_in	2023-02-14 01:57:40 +00:00
Woosuk Kwon	531e1c74e8	Fix typo	2023-02-13 18:51:33 +00:00
Woosuk Kwon	d3d317665b	Fix scheduler	2023-02-13 09:37:00 +00:00
Woosuk Kwon	5a309bb588	Add scheduler	2023-02-13 02:39:53 +00:00
Woosuk Kwon	eb52db1bea	Fix can_swap_in	2023-02-13 02:38:50 +00:00
Woosuk Kwon	a2a9869cb7	SERVING -> RUNNING	2023-02-12 08:25:05 +00:00
Woosuk Kwon	46958cf941	BlockAllocator -> BlockManager	2023-02-12 08:15:18 +00:00
Woosuk Kwon	38ed06a846	Minor	2023-02-09 11:30:55 +00:00
Woosuk Kwon	aa78aeaa0f	Add block manager	2023-02-09 11:27:33 +00:00