72 Commits

Author	SHA1	Message	Date
Woosuk Kwon	e9d3f2ff77	Add memory analyzer & utomatically configure KV cache size (#6)	2023-03-11 23:23:14 -08:00
Woosuk Kwon	1a7eb7da61	Support beam search & parallel generation (#7)	2023-03-10 09:58:21 -08:00
Woosuk Kwon	04e5acc08e	Fix a bug in 1D input shape (#5)	2023-03-06 10:05:27 -08:00
Woosuk Kwon	3e9f991d6a	Use FlashAttention for multi_query_kv_attention (#4)	2023-03-01 21:13:08 -08:00
Woosuk Kwon	0deacbce6e	Implement single_query_cached_kv_attention kernel (#3)	2023-03-01 15:02:19 -08:00
Woosuk Kwon	cbf8779afa	Fix a bug in tying OPT embeddings (#1)	2023-02-24 16:29:36 -08:00
Woosuk Kwon	6aef2278f4	[Minor] Fix printing format	2023-02-24 11:56:06 +00:00
Woosuk Kwon	1132fae0ca	Add Frontend	2023-02-24 11:46:43 +00:00
Woosuk Kwon	46ce1356f7	Add max_num_steps to SamplingParams	2023-02-24 11:44:40 +00:00
Woosuk Kwon	b39f149a08	Add is_finished	2023-02-24 11:44:21 +00:00
Woosuk Kwon	ef6098ec51	Merge pre_step and step	2023-02-24 10:36:08 +00:00
Woosuk Kwon	53f70e7334	Reduce the number of states in scheduler	2023-02-24 10:22:39 +00:00
Woosuk Kwon	762fd1c3fa	Refactor and annotate types for attention	2023-02-24 08:58:46 +00:00
Woosuk Kwon	7f22f90e8c	Remove xformers	2023-02-24 08:36:16 +00:00
Woosuk Kwon	afdbe5d373	[WIP] Add server script	2023-02-24 01:33:37 +00:00
Woosuk Kwon	932844f1cd	Fix attention	2023-02-23 23:02:25 +00:00
Woosuk Kwon	ba84b8728a	Fix attention	2023-02-23 22:29:46 +00:00
Woosuk Kwon	87e0bcd426	Fix attention	2023-02-23 21:32:02 +00:00
Woosuk Kwon	1ce1333573	Set default dtype to half	2023-02-23 21:31:39 +00:00
Woosuk Kwon	de0fabbc5c	Fix sampler	2023-02-23 20:30:12 +00:00
Woosuk Kwon	fdd0f2f472	Minor	2023-02-23 20:23:47 +00:00
Woosuk Kwon	7f985166f7	Consider pempty tensor	2023-02-23 20:20:33 +00:00
Woosuk Kwon	86f9eb6d39	Fix typo	2023-02-23 20:19:24 +00:00
Woosuk Kwon	1f6c7ef437	Add controller	2023-02-23 09:32:19 +00:00
Woosuk Kwon	d4bc1a4d24	Add unoptimized OPT Attention	2023-02-23 09:31:55 +00:00
Woosuk Kwon	b56b6ca0d6	Add greedy sampler	2023-02-23 09:26:09 +00:00
Woosuk Kwon	343cea3dbc	Add seq_ids to input metadata	2023-02-23 09:25:01 +00:00
Woosuk Kwon	4f6f4967f6	Add get_block_table	2023-02-23 07:55:14 +00:00
Woosuk Kwon	331fa0b042	Implement scheduler.step & Add a threshold for batch size	2023-02-23 07:54:20 +00:00
Woosuk Kwon	501c4bd0cd	decoding.py -> sampling_params.py	2023-02-23 07:39:20 +00:00
Woosuk Kwon	86c682cd32	DecodingParams -> SamplingParams	2023-02-23 07:38:43 +00:00
Woosuk Kwon	af16c05074	Add get_len	2023-02-23 05:58:04 +00:00
Woosuk Kwon	d094512296	Move max_context_len	2023-02-23 04:57:46 +00:00
Woosuk Kwon	4b1ac23f53	Fix slot mapping	2023-02-23 00:10:07 +00:00
Woosuk Kwon	8290fce47d	Add Worker class	2023-02-22 19:01:38 +00:00
Woosuk Kwon	7b6844e590	Add input metadata	2023-02-22 19:01:20 +00:00
Woosuk Kwon	608f74ffe5	Minor	2023-02-22 18:08:25 +00:00
Woosuk Kwon	709a69176e	Move worker/models -> models	2023-02-22 18:03:48 +00:00
Woosuk Kwon	6f058c7ba8	Implement cache ops	2023-02-16 07:47:03 +00:00
Woosuk Kwon	a1c67e6db8	Minor	2023-02-16 01:42:53 +00:00
Woosuk Kwon	9e68a6827e	Fix return type error	2023-02-16 01:33:03 +00:00
Woosuk Kwon	8edcabc737	Add warning	2023-02-16 01:28:17 +00:00
Woosuk Kwon	2f4887de77	Fix KVCache shape	2023-02-16 01:24:45 +00:00
Woosuk Kwon	3363c27d19	Add __repr__	2023-02-14 09:34:07 +00:00
Woosuk Kwon	2729087efe	Fix a ref count bug in swapping	2023-02-14 09:20:12 +00:00
Woosuk Kwon	c128c2ed03	Fix double free	2023-02-14 09:14:05 +00:00
Woosuk Kwon	c80ed212a1	Minor	2023-02-14 09:06:27 +00:00
Woosuk Kwon	263e91b386	Minor	2023-02-14 09:05:39 +00:00
Woosuk Kwon	be1e2163c9	Fix memory leak in swapping	2023-02-14 08:46:33 +00:00
Woosuk Kwon	7e5f604e68	Fix bugs in scheduler	2023-02-14 02:25:32 +00:00