# nano-X1 Tokenizer

ByteLevel BPE tokenizer based on EleutherAI/gpt-neox-20b with ChatML + `<think>`/`</think>` special tokens added.

Load: `AutoTokenizer.from_pretrained('./nano_v5_tok_nanox1')`