Harry Mellor fecae12cd7

Remove all_special_tokens_extended from tokenizer code (#29686) ...

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

2025-11-28 20:26:51 +00:00

..

__init__.py

…

test_cached_tokenizer.py

Remove all_special_tokens_extended from tokenizer code (#29686)

2025-11-28 20:26:51 +00:00

test_detokenize.py

…

test_do_lower_case.py

…

test_get_eos.py

…

test_mistral_tokenizer.py

Remove all_special_tokens_extended from tokenizer code (#29686)

2025-11-28 20:26:51 +00:00

test_tokenizer_registry.py

Remove all_special_tokens_extended from tokenizer code (#29686)

2025-11-28 20:26:51 +00:00

test_tokenizer.py

Convert formatting to use ruff instead of yapf + isort (#26247)

2025-10-05 07:06:22 -07:00