xinyun/vllm
1
0
Fork 0
mirror of https://git.datalinker.icu/vllm-project/vllm.git synced 2026-09-09 07:07:01 +08:00
Code Issues Packages Projects Releases Wiki Activity

Fill TorchSDPAAttentionMetadata seq_lens_field for prefill (#10799)

Browse Source 
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
This commit is contained in:
Maximilien de Bayser 2024-12-01 23:05:32 -03:00 committed by GitHub
parent 073a4bd1c0
commit e25810ae29
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 5 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
vllm/attention/backends/torch_sdpa.py
View File

@ -341,7 +341,11 @@ class TorchSDPAMetadataBuilder(AttentionMetadataBuilder[TorchSDPAMetadata]):
) )
else: else:
block_tables = torch.tensor([]) block_tables = torch.tensor([])
seq_lens_tensor = torch.tensor([]) seq_lens_tensor = torch.tensor(
input_data.seq_lens[:input_data.num_prefills],
dtype=torch.int32,
device="cpu",
)
# For multi-modal models # For multi-modal models
placeholder_index_maps = None placeholder_index_maps = None