ray0rf1re/Nano-Nano_v5.1
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity
Files
main
Nano-Nano_v5.1/tokenizer/README.md
ModelHub XC 2f5adbaa8b 初始化项目，由ModelHub XC社区提供模型 
Model: ray0rf1re/Nano-Nano_v5.1
Source: Original Platform
2026-07-15 21:56:43 +08:00

198 B
Raw Permalink Blame History

nano-X1 Tokenizer

ByteLevel BPE tokenizer based on EleutherAI/gpt-neox-20b with ChatML + <think>/</think> special tokens added.

Load: AutoTokenizer.from_pretrained('./nano_v5_tok_nanox1')