Nick Hill e46a60aa4c

[BugFix] Fix handling of stop strings and stop token ids (#3672)

2024-04-11 15:34:12 -07:00

..

configs

[Model] Add support for DBRX (#3660)

2024-03-27 13:01:46 -07:00

tokenizer_group

[BugFix] Pass tokenizer_config to local_tokenizer_group (#3754)

2024-04-03 20:31:46 -07:00

tokenizers

Re-enable the 80 char line width limit (#3305)

2024-03-10 19:49:14 -07:00

__init__.py

[Tokenizer] Add an option to specify tokenizer (#284)

2023-06-28 09:46:58 -07:00

config.py

[Model] Add support for DBRX (#3660)

2024-03-27 13:01:46 -07:00

detokenizer.py

[BugFix] Fix handling of stop strings and stop token ids (#3672)

2024-04-11 15:34:12 -07:00

tokenizer.py

[Misc] Some minor simplifications to detokenization logic (#3670)

2024-04-01 13:22:06 -07:00