Tyler Michael Smith 8d75fe48ca

[Kernel] Switch fp8 layers to use the CUTLASS kernels (#5183) ...

Switching from torch._scaled_mm to vLLM's cutlass fp8 kernels when supported as we are seeing 5-15% improvement in e2e performance on neuralmagic/Meta-Llama-3-8B-Instruct-FP8

see https://docs.google.com/spreadsheets/d/1GiAnmzyGHgZ6zL_LDSTm35Bdrt4A8AaFEurDlISYYA4/ for some quick e2e benchmarks and #5144 for comparisons across different GEMM sizes.

2024-06-07 08:42:35 +00:00

..

compressed_tensors

[Kernel] Pass a device pointer into the quantize kernel for the scales (#5159)

2024-06-03 09:52:30 -07:00

utils

[Kernel] Add marlin_24 unit tests (#4901)

2024-05-19 11:37:34 -04:00

__init__.py

[Feature][Kernel] Support bitsandbytes quantization and QLoRA (#4776)

2024-06-01 14:51:10 -06:00

aqlm.py

[Core] Allow AQLM on Pascal (#5058)

2024-05-27 15:26:14 -07:00

awq.py

[Misc][Refactor] Generalize linear_method to be quant_method (#4373)

2024-04-26 16:41:14 -04:00

base_config.py

Add GPTQ Marlin 2:4 sparse structured support (#4790)

2024-05-16 12:56:15 -04:00

bitsandbytes.py

[Feature][Kernel] Support bitsandbytes quantization and QLoRA (#4776)

2024-06-01 14:51:10 -06:00

deepspeedfp.py

[Model] Snowflake arctic model implementation (#4652)

2024-05-09 22:37:14 +00:00

fp8.py

[Kernel] Switch fp8 layers to use the CUTLASS kernels (#5183)

2024-06-07 08:42:35 +00:00

gptq_marlin_24.py

Marlin 24 prefill performance improvement (about 25% better on average) (#4983)

2024-05-23 02:39:27 -04:00

gptq_marlin.py

[Bugfix] gptq_marlin: Ensure g_idx_sort_indices is not a Parameter (#5108)

2024-05-30 00:30:18 +00:00

gptq.py

[Misc][Refactor] Generalize linear_method to be quant_method (#4373)

2024-04-26 16:41:14 -04:00

marlin.py

Add GPTQ Marlin 2:4 sparse structured support (#4790)

2024-05-16 12:56:15 -04:00

schema.py

Enable scaled FP8 (e4m3fn) KV cache on ROCm (AMD GPU) (#3290)

2024-04-03 14:15:55 -07:00

squeezellm.py

[mypy][5/N] Support all typing on model executor (#4427)

2024-04-28 19:01:26 -07:00