Lucas Wilkinson baeded2569

[Attention] Deepseek v3 MLA support with FP8 compute (#12601) ...

This PR implements the Deepseek V3 support by performing matrix absorption the fp8 weights 

---------

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: simon-mo <simon.mo@hey.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
Co-authored-by: Tyler Michael Smith <tysmith@redhat.com>
Co-authored-by: Alexander Matveev <59768536+alexm-neuralmagic@users.noreply.github.com>

2025-01-31 21:52:51 -08:00

..

__init__.py

[Misc] Rename embedding classes to pooling (#10801)

2024-12-01 14:36:51 +08:00

adapters.py

[Model] Automatic conversion of classification and reward models (#11469)

2024-12-24 18:22:22 +00:00

arctic.py

[model][utils] add extract_layer_index utility function (#10599)

2024-11-23 22:22:54 -08:00

aria.py

[Model][Bugfix]: correct Aria model output (#12309)

2025-01-22 11:39:19 +00:00

baichuan.py

[Misc] Merge bitsandbytes_stacked_params_mapping and packed_modules_mapping (#11924)

2025-01-15 07:49:49 +08:00

bart.py

[Kernel] Move attn_type to Attention.__init__() (#11690)

2025-01-07 00:11:28 +08:00

bert.py

[Kernel] Move attn_type to Attention.__init__() (#11690)

2025-01-07 00:11:28 +08:00

blip2.py

[Bugfix] Fix BLIP-2 processing (#12412)

2025-01-25 11:42:42 +08:00

blip.py

[V1][VLM] V1 support for selected single-image models. (#11632)

2024-12-31 21:17:22 +00:00

bloom.py

[torch.compile] support all attention backends (#10558)

2024-11-22 14:04:42 -08:00

chameleon.py

[VLM] Avoid unnecessary tokenization (#12310)

2025-01-22 11:08:31 +00:00

chatglm.py

[Misc] Add BNB support to GLM4-V model (#12184)

2025-01-19 19:49:22 +08:00

clip.py

[Misc] Move some model utils into vision file (#11848)

2025-01-08 17:04:46 +00:00

commandr.py

Various cosmetic/comment fixes (#12089)

2025-01-16 09:59:06 +00:00

dbrx.py

Various cosmetic/comment fixes (#12089)

2025-01-16 09:59:06 +00:00

decilm.py

[Misc] Add uninitialized params tracking for AutoWeightsLoader (#10327)

2024-11-18 09:07:46 +08:00

deepseek_v2.py

[Attention] MLA decode optimizations (#12528)

2025-01-30 23:49:37 -08:00

deepseek_v3.py

[Attention] Deepseek v3 MLA support with FP8 compute (#12601)

2025-01-31 21:52:51 -08:00

deepseek_vl2.py

[VLM] Avoid unnecessary tokenization (#12310)

2025-01-22 11:08:31 +00:00

deepseek.py

[model][utils] add extract_layer_index utility function (#10599)

2024-11-23 22:22:54 -08:00

eagle.py

[CI][Spec Decode] fix: broken test for EAGLE model (#11972)

2025-01-13 06:50:35 +00:00

exaone.py

[FP8][Kernel] Dynamic kv cache scaling factors computation (#11906)

2025-01-23 18:04:03 +00:00

fairseq2_llama.py

[Model] Support for fairseq2 Llama (#11442)

2025-01-19 10:40:40 -08:00

falcon.py

[Misc] Merge bitsandbytes_stacked_params_mapping and packed_modules_mapping (#11924)

2025-01-15 07:49:49 +08:00

florence2.py

[torch.compile] support all attention backends (#10558)

2024-11-22 14:04:42 -08:00

fuyu.py

[VLM] Avoid unnecessary tokenization (#12310)

2025-01-22 11:08:31 +00:00

gemma2.py

[Misc][Quark] Upstream Quark format to VLLM (#10765)

2025-01-15 11:05:15 -05:00

gemma.py

Update pre-commit hooks (#12475)

2025-01-27 17:23:08 -07:00

glm4_vision_encoder.py

[Misc] Add BNB support to GLM4-V model (#12184)

2025-01-19 19:49:22 +08:00

glm.py

[Model] Added GLM-4 series hf format model support vllm==0.6.4 (#10561)

2024-11-28 14:53:31 +00:00

gpt2.py

[Bugfix] Fix gpt2 GGUF inference (#12467)

2025-01-27 18:31:49 +08:00

gpt_bigcode.py

[torch.compile] support all attention backends (#10558)

2024-11-22 14:04:42 -08:00

gpt_j.py

Various cosmetic/comment fixes (#12089)

2025-01-16 09:59:06 +00:00

gpt_neox.py

[torch.compile] support all attention backends (#10558)

2024-11-22 14:04:42 -08:00

granite.py

[FP8][Kernel] Dynamic kv cache scaling factors computation (#11906)

2025-01-23 18:04:03 +00:00

granitemoe.py

Update pre-commit hooks (#12475)

2025-01-27 17:23:08 -07:00

gritlm.py

[Frontend] Separate pooling APIs in offline inference (#11129)

2024-12-13 10:40:07 +00:00

h2ovl.py

[1/N] Initial prototype for multi-modal processor (#10044)

2024-11-13 12:39:03 +00:00

idefics2_vision_model.py

[V1][VLM] V1 support for selected single-image models. (#11632)

2024-12-31 21:17:22 +00:00

idefics3.py

[Misc] Merge bitsandbytes_stacked_params_mapping and packed_modules_mapping (#11924)

2025-01-15 07:49:49 +08:00

interfaces_base.py

[Misc] Remove redundant TypeVar from base model (#12248)

2025-01-21 08:40:39 +00:00

interfaces.py

[VLM] Enable tokenized inputs for merged multi-modal processor (#11900)

2025-01-10 03:24:00 +00:00

intern_vit.py

[Bug] Fix usage of .transpose() and .view() consecutively. (#11979)

2025-01-13 06:24:10 +00:00

internlm2_ve.py

[torch.compile] support all attention backends (#10558)

2024-11-22 14:04:42 -08:00

internlm2.py

[Model] Support InternLM2 Reward models (#11571)

2024-12-28 06:14:10 +00:00

internvl.py

[V1][VLM] Fix edge case bug for InternVL2 (#11165)

2024-12-13 11:09:30 +00:00

jais.py

[torch.compile] support all attention backends (#10558)

2024-11-22 14:04:42 -08:00

jamba.py

[MODEL] LoRA support for Jamba model (#11209)

2024-12-27 17:58:21 +00:00

llama.py

[Hardware][NV] Fix Modelopt model loading for k-v-scales for Llama models. (#11787)

2025-01-29 01:46:12 -08:00

llava_next_video.py

[VLM] Reorganize profiling/processing-related code (#11812)

2025-01-08 18:59:58 +08:00

llava_next.py

[VLM] Reorganize profiling/processing-related code (#11812)

2025-01-08 18:59:58 +08:00

llava_onevision.py

[Bugfix][VLM] Fix mixed-modality inference backward compatibility for V0 (#12313)

2025-01-22 21:06:36 +08:00

llava.py

[VLM] Avoid unnecessary tokenization (#12310)

2025-01-22 11:08:31 +00:00

mamba_cache.py

[Kernel][Model] Improve continuous batching for Jamba and Mamba (#9189)

2024-10-16 12:12:43 -04:00

mamba.py

[MODEL] LoRA support for Jamba model (#11209)

2024-12-27 17:58:21 +00:00

medusa.py

[model] Reduce medusa weight (#10454)

2024-11-20 06:05:55 +00:00

minicpm3.py

[Misc] Merge bitsandbytes_stacked_params_mapping and packed_modules_mapping (#11924)

2025-01-15 07:49:49 +08:00

minicpm.py

[Misc] Merge bitsandbytes_stacked_params_mapping and packed_modules_mapping (#11924)

2025-01-15 07:49:49 +08:00

minicpmo.py

[Model] Refactoring of MiniCPM-V and add MiniCPM-o-2.6 support for vLLM (#12069)

2025-01-29 09:24:59 +00:00

minicpmv.py

[Model] Refactoring of MiniCPM-V and add MiniCPM-o-2.6 support for vLLM (#12069)

2025-01-29 09:24:59 +00:00

mixtral_quant.py

[torch.compile] support all attention backends (#10558)

2024-11-22 14:04:42 -08:00

mixtral.py

[Hardware][NV] Fix Modelopt model loading for k-v-scales for Llama models. (#11787)

2025-01-29 01:46:12 -08:00

mllama.py

[ROCm][AMD][Model] llama 3.2 support upstreaming (#12421)

2025-01-31 12:24:28 +08:00

mlp_speculator.py

Update pre-commit hooks (#12475)

2025-01-27 17:23:08 -07:00

module_mapping.py

[Model][LoRA]LoRA support added for MiniCPMV2.5 (#7199)

2024-09-29 06:59:45 +00:00

molmo.py

[Kernel] Support MulAndSilu (#11624)

2025-01-15 02:29:53 +00:00

mpt.py

[torch.compile] support all attention backends (#10558)

2024-11-22 14:04:42 -08:00

nemotron.py

Various cosmetic/comment fixes (#12089)

2025-01-16 09:59:06 +00:00

nvlm_d.py

[Bugfix] Fix _init_vision_model in NVLM_D model (#9611)

2024-10-23 14:09:04 +00:00

olmo2.py

[Model] Add OLMo November 2024 model (#10503)

2024-11-25 17:26:40 -05:00

olmo.py

[torch.compile] support all attention backends (#10558)

2024-11-22 14:04:42 -08:00

olmoe.py

[Misc] Move print_*_once from utils to logger (#11298)

2025-01-09 12:48:12 +08:00

opt.py

[Misc] Merge bitsandbytes_stacked_params_mapping and packed_modules_mapping (#11924)

2025-01-15 07:49:49 +08:00

orion.py

[torch.compile] support all attention backends (#10558)

2024-11-22 14:04:42 -08:00

paligemma.py

[ci/lint] Add back default arg for pre-commit (#12279)

2025-01-22 01:15:27 +00:00

persimmon.py

[torch.compile] support all attention backends (#10558)

2024-11-22 14:04:42 -08:00

phi3_small.py

[platform] support custom torch.compile backend key (#11318)

2025-01-10 23:46:51 +08:00

phi3.py

[Misc] Merge bitsandbytes_stacked_params_mapping and packed_modules_mapping (#11924)

2025-01-15 07:49:49 +08:00

phi3v.py

[VLM] Simplify post-processing of replacement info (#12269)

2025-01-21 16:48:13 -08:00

phi.py

[Misc] Merge bitsandbytes_stacked_params_mapping and packed_modules_mapping (#11924)

2025-01-15 07:49:49 +08:00

phimoe.py

Update pre-commit hooks (#12475)

2025-01-27 17:23:08 -07:00

pixtral.py

[Misc] Move some model utils into vision file (#11848)

2025-01-08 17:04:46 +00:00

qwen2_audio.py

[VLM] Avoid unnecessary tokenization (#12310)

2025-01-22 11:08:31 +00:00

qwen2_moe.py

[Misc] Move print_*_once from utils to logger (#11298)

2025-01-09 12:48:12 +08:00

qwen2_rm.py

[Model] Add Qwen2 PRM model support (#12202)

2025-01-20 14:59:46 +08:00

qwen2_vl.py

[Bugfix][VLM] Fix mixed-modality inference backward compatibility for V0 (#12313)

2025-01-22 21:06:36 +08:00

qwen2.py

[V1] Add V1 support of Qwen2-VL (#12128)

2025-01-19 19:52:13 +08:00

qwen.py

[VLM] Merged multi-modal processor and V1 support for Qwen-VL (#12504)

2025-01-28 16:25:05 +00:00

registry.py

[Model] Refactoring of MiniCPM-V and add MiniCPM-o-2.6 support for vLLM (#12069)

2025-01-29 09:24:59 +00:00

roberta.py

[Bugfix] Fix RobertaModel loading (#11940)

2025-01-11 14:05:09 +00:00

siglip.py

[ci/lint] Add back default arg for pre-commit (#12279)

2025-01-22 01:15:27 +00:00

solar.py

[FP8][Kernel] Dynamic kv cache scaling factors computation (#11906)

2025-01-23 18:04:03 +00:00

stablelm.py

[Model] Support GGUF models newly added in transformers 4.46.0 (#9685)

2025-01-13 00:13:44 +00:00

starcoder2.py

[Model] Support GGUF models newly added in transformers 4.46.0 (#9685)

2025-01-13 00:13:44 +00:00

telechat2.py

[Bugfix] Fix TeleChat2ForCausalLM weights mapper (#11546)

2024-12-27 10:39:15 +00:00

ultravox.py

Update pre-commit hooks (#12475)

2025-01-27 17:23:08 -07:00

utils.py

Update pre-commit hooks (#12475)

2025-01-27 17:23:08 -07:00

vision.py

[Misc] Set default backend to SDPA for get_vit_attn_backend (#12235)

2025-01-21 11:52:11 -08:00

whisper.py

Bugfix for whisper quantization due to fake k_proj bias (#12524)

2025-01-29 04:49:03 +00:00