2 Commits

Author SHA1 Message Date
majiayu000
abd1dbc548 [Bugfix] Preserve original tokenizer class name in CachedTokenizer
HuggingFace transformers processor validates tokenizer type by checking
the class name. When vLLM creates a CachedTokenizer with a modified
class name (e.g., 'CachedQwen2TokenizerFast'), the processor type check
fails with TypeError.

This fix preserves the original tokenizer class name and qualname in
CachedTokenizer, ensuring compatibility with HuggingFace transformers
processor type checking.

Fixes #31080

Signed-off-by: Claude <noreply@anthropic.com>
Signed-off-by: majiayu000 <1835304752@qq.com>
2025-12-24 16:02:48 +08:00
Cyrus Leung
2afcec4dec
[Misc] Update TokenizerLike interface and move get_cached_tokenizer (#29730)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2025-11-30 14:59:47 +08:00