6 lines
198 B
Markdown
6 lines
198 B
Markdown
|
# nano-X1 Tokenizer
|
||
|
|
|
||
|
|
ByteLevel BPE tokenizer based on EleutherAI/gpt-neox-20b with ChatML + `<think>`/`</think>` special tokens added.
|
||
|
|
|
||
|
|
Load: `AutoTokenizer.from_pretrained('./nano_v5_tok_nanox1')`
|