fix(tokenizer): Sets fim and think tokens to non-special, and removes unk_token.

Browse Source

...

This commit is contained in:

ai-modelscope

2025-07-23 00:34:30 +08:00

parent 0bb594bd98

commit cd9c55560e

3 changed files with 71 additions and 71 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files