Files
gpt2-small-italian/tokenizer_config.json

6 lines
135 B
JSON
Raw Permalink Normal View History

{
"unk_token": "<|endoftext|>",
"bos_token": "<|endoftext|>",
"eos_token": "<|endoftext|>",
"add_prefix_space": false
}