vllm/examples at fdea8ec16775e1645620b5ff46b799d60df4624c - vllm

chart-helm

Add example of helm chart for vllm deployment on k8s (#9199 )

2024-12-10 09:19:27 +00:00

fp8

Adds method to read the pooling types from model's files (#9506 )

2024-11-07 08:42:40 +00:00

production_monitoring

[Metrics] add more metrics (#4464 )

2024-11-12 00:17:38 +08:00

api_client.py

…

aqlm_example.py

…

cpu_offload.py

…

disaggregated_prefill.sh

[Core] Implement disagg prefill by StatelessProcessGroup (#10502 )

2024-12-01 19:01:00 -06:00

florence2_inference.py

[Model] Initialize Florence-2 language backbone support (#9555 )

2024-10-23 10:42:47 +00:00

gguf_inference.py

…

gradio_openai_chatbot_webserver.py

…

gradio_webserver.py

…

llm_engine_example.py

[core] remove beam search from the core (#9105 )

2024-10-07 05:47:04 +00:00

logging_configuration.md

[Core] Fix broken log configuration (#10458 )

2024-11-23 10:23:51 +08:00

lora_with_quantization_inference.py

[[Misc]Upgrade bitsandbytes to the latest version 0.44.0 (#8768 )

2024-09-24 17:08:55 -07:00

multilora_inference.py

[core] remove beam search from the core (#9105 )

2024-10-07 05:47:04 +00:00

offline_chat_with_tools.py

[Model] Add mistral function calling format to all models loaded with "mistral" format (#8515 )

2024-09-17 17:50:37 +00:00

offline_inference_arctic.py

…

offline_inference_audio_language.py

[Model] Refactor Ultravox to use merged input processor (#11198 )

2024-12-16 10:09:53 +00:00

offline_inference_chat.py

[Frontend] Batch inference for llm.chat() API (#8648 )

2024-09-24 09:44:11 -07:00

offline_inference_classification.py

[Frontend] Separate pooling APIs in offline inference (#11129 )

2024-12-13 10:40:07 +00:00

offline_inference_cli.py

[Minor] Revert change in offline inference example (#10545 )

2024-11-21 21:28:16 +00:00

offline_inference_distributed.py

…

offline_inference_embedding.py

[Frontend] Separate pooling APIs in offline inference (#11129 )

2024-12-13 10:40:07 +00:00

offline_inference_encoder_decoder.py

[Core] Support serving encoder/decoder models (#7258 )

2024-08-09 10:39:41 +08:00

offline_inference_mlpspeculator.py

[Core] Deprecating block manager v1 and make block manager v2 default (#8704 )

2024-10-17 11:38:15 -05:00

offline_inference_neuron_int8_quantization.py

[Neuron] Adding support for adding/ overriding neuron configuration a… (#8062 )

2024-09-04 16:33:43 -07:00

offline_inference_neuron.py

[Neuron] Adding support for context-lenght, token-gen buckets. (#7885 )

2024-08-29 13:58:14 -07:00

offline_inference_openai.md

[Doc] Reorganize online pooling APIs (#11172 )

2024-12-14 00:22:22 +08:00

offline_inference_pixtral.py

[Misc] Update Pixtral example (#8431 )

2024-09-12 17:31:18 -07:00

offline_inference_scoring.py

[Frontend] Separate pooling APIs in offline inference (#11129 )

2024-12-13 10:40:07 +00:00

offline_inference_structured_outputs.py

[Doc] Add documentation for Structured Outputs (#9943 )

2024-11-18 09:52:12 -08:00

offline_inference_tpu.py

…

offline_inference_vision_language_embedding.py

[Frontend] Separate pooling APIs in offline inference (#11129 )

2024-12-13 10:40:07 +00:00

offline_inference_vision_language_multi_image.py

[Doc] Reorder vision language examples in alphabet order (#11228 )

2024-12-16 11:23:33 +00:00

offline_inference_vision_language.py

[V1] VLM - enable processor cache by default (#11305 )

2024-12-18 18:54:46 -05:00

offline_inference_with_prefix.py

[MISC] Consolidate cleanup() and refactor offline_inference_with_prefix.py (#9510 )

2024-10-18 14:30:55 -07:00

offline_inference_with_profiler.py

[V1] Fix torch profiling for offline inference (#11125 )

2024-12-12 15:51:53 +00:00

offline_inference.py

[Feature] vLLM ARM Enablement for AARCH64 CPUs (#9228 )

2024-11-25 18:32:39 -08:00

offline_profile.py

[misc] Layerwise profile updates (#10242 )

2024-12-16 18:14:57 +00:00

openai_chat_completion_client_for_multimodal.py

[Frontend] Add OpenAI API support for input_audio (#11027 )

2024-12-16 22:09:58 -08:00

openai_chat_completion_client_with_tools.py

[Feature] OpenAI-Compatible Tools API + Streaming for Hermes & Mistral models (#5649 )

2024-09-04 13:18:13 -07:00

openai_chat_completion_client.py

…

openai_chat_completion_structured_outputs.py

[Doc] Add documentation for Structured Outputs (#9943 )

2024-11-18 09:52:12 -08:00

openai_chat_embedding_client_for_multimodal.py

[Doc] Reorganize online pooling APIs (#11172 )

2024-12-14 00:22:22 +08:00

openai_completion_client.py

…

openai_cross_encoder_score.py

[Doc] Reorganize online pooling APIs (#11172 )

2024-12-14 00:22:22 +08:00

openai_embedding_client.py

[Bugfix]: Use float32 for base64 embedding (#7855 )

2024-08-26 03:16:38 +00:00

openai_example_batch.jsonl

[Misc][OpenAI] deprecate max_tokens in favor of new max_completion_tokens field for chat completion endpoint (#9837 )

2024-10-30 18:15:56 -07:00

run_cluster.sh

[CI/Build] Add shell script linting using shellcheck (#7925 )

2024-11-07 18:17:29 +00:00

save_sharded_state.py

…

template_alpaca.jinja

…

template_baichuan.jinja

…

template_blip2.jinja

…

template_chatglm2.jinja

…

template_chatglm.jinja

…

template_chatml.jinja

…

template_dse_qwen2_vl.jinja

[Model] Adding Support for Qwen2VL as an Embedding Model. Using MrLight/dse-qwen2-2b-mrl-v1 (#9944 )

2024-11-13 08:28:13 +00:00

template_falcon_180b.jinja

…

template_falcon.jinja

…

template_inkbot.jinja

…

template_llava.jinja

…

template_vlm2vec.jinja

[Frontend] Use a proper chat template for VLM2Vec (#9912 )

2024-11-01 14:09:07 +00:00

tensorize_vllm_model.py

…

tool_chat_template_granite_20b_fc.jinja

[Model] tool calling support for ibm-granite/granite-20b-functioncalling (#8339 )

2024-10-29 15:07:37 -07:00

tool_chat_template_granite.jinja

Change granite chat template to keep json list formatting for tool calls (#10452 )

2024-11-19 18:16:54 -07:00

tool_chat_template_hermes.jinja

[Bugfix] Fix Hermes tool call chat template bug (#8256 )

2024-09-07 10:49:01 +08:00

tool_chat_template_internlm2_tool.jinja

[Frontend][Feature] support tool calling for internlm/internlm2_5-7b-chat model (#8405 )

2024-10-04 10:36:39 +08:00

tool_chat_template_llama3.1_json.jinja

[Bugfix][Frontend] Update Llama Chat Templates to also support Non-Tool use (#10164 )

2024-11-23 10:17:38 +08:00

tool_chat_template_llama3.2_json.jinja

[Misc] Update llama 3.2 template to support system prompt with images (#10901 )

2024-12-05 05:54:06 +00:00

tool_chat_template_llama3.2_pythonic.jinja

[Frontend] Pythonic tool parser (#9859 )

2024-11-14 04:14:34 +00:00

tool_chat_template_mistral_parallel.jinja

[Bugfix] example template should not add parallel_tool_prompt if tools is none (#9007 )

2024-10-03 03:04:17 +00:00

tool_chat_template_mistral.jinja

[Feature] OpenAI-Compatible Tools API + Streaming for Hermes & Mistral models (#5649 )

2024-09-04 13:18:13 -07:00

tool_chat_template_toolace.jinja

[Frontend] Pythonic tool parser (#9859 )

2024-11-14 04:14:34 +00:00