12 Commits

Author	SHA1	Message	Date
Woosuk Kwon	e9d3f2ff77	Add memory analyzer & utomatically configure KV cache size (#6)	2023-03-11 23:23:14 -08:00
Woosuk Kwon	1a7eb7da61	Support beam search & parallel generation (#7)	2023-03-10 09:58:21 -08:00
Woosuk Kwon	1132fae0ca	Add Frontend	2023-02-24 11:46:43 +00:00
Woosuk Kwon	ef6098ec51	Merge pre_step and step	2023-02-24 10:36:08 +00:00
Woosuk Kwon	53f70e7334	Reduce the number of states in scheduler	2023-02-24 10:22:39 +00:00
Woosuk Kwon	afdbe5d373	[WIP] Add server script	2023-02-24 01:33:37 +00:00
Woosuk Kwon	fdd0f2f472	Minor	2023-02-23 20:23:47 +00:00
Woosuk Kwon	331fa0b042	Implement scheduler.step & Add a threshold for batch size	2023-02-23 07:54:20 +00:00
Woosuk Kwon	7e5f604e68	Fix bugs in scheduler	2023-02-14 02:25:32 +00:00
Woosuk Kwon	531e1c74e8	Fix typo	2023-02-13 18:51:33 +00:00
Woosuk Kwon	d3d317665b	Fix scheduler	2023-02-13 09:37:00 +00:00
Woosuk Kwon	5a309bb588	Add scheduler	2023-02-13 02:39:53 +00:00