Cyrus Leung b4e9fd811f

Revert "[PERF] Use faster way of decode in tokenizer: avoid useless list-to-list conversion (#20000)" (#23396) ...

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

2025-08-23 04:16:48 +00:00

..

chat_templates

[Misc] Add SPDX-FileCopyrightText (#19100)

2025-06-03 11:20:17 -07:00

configs

[Bugfix] Fix extra whitespace in strings caused by newline (#23272)

2025-08-20 22:03:00 -07:00

processors

[Model] support new model ovis2.5 (#23084)

2025-08-19 13:12:59 +00:00

tokenizers

[Bugfix] Replace custom Encoding class with BatchEncoding in MistralTokenizer (#22786)

2025-08-14 08:09:27 -07:00

__init__.py

[Misc] Add SPDX-FileCopyrightText (#19100)

2025-06-03 11:20:17 -07:00

config.py

[Fix] fix offline env use local mode path (#22526)

2025-08-20 13:34:49 +00:00

detokenizer_utils.py

[Perf] Speed up function _convert_tokens_to_string_with_added_encoders by 13.7x (#20413)

2025-08-20 13:17:11 -07:00

detokenizer.py

[Misc] Add SPDX-FileCopyrightText (#19100)

2025-06-03 11:20:17 -07:00

dynamic_module.py

[Deprecation][2/N] Replace --task with --runner and --convert (#21470)

2025-07-27 19:42:40 -07:00

processor.py

[Misc] Automatically resolve HF processor init kwargs (#22005)

2025-07-31 22:44:10 -07:00

s3_utils.py

[Misc] Add SPDX-FileCopyrightText (#19100)

2025-06-03 11:20:17 -07:00

tokenizer_base.py

[Misc] Add SPDX-FileCopyrightText (#19100)

2025-06-03 11:20:17 -07:00

tokenizer_group.py

[Deprecation][2/N] Replace --task with --runner and --convert (#21470)

2025-07-27 19:42:40 -07:00

tokenizer.py

Revert "[PERF] Use faster way of decode in tokenizer: avoid useless list-to-list conversion (#20000)" (#23396)

2025-08-23 04:16:48 +00:00

utils.py

[Misc] Add SPDX-FileCopyrightText (#19100)

2025-06-03 11:20:17 -07:00