Files
gpt2-medium-italian-embeddings/tokenizer_config.json

6 lines
135 B
JSON
Raw Permalink Normal View History

{
"unk_token": "<|endoftext|>",
"bos_token": "<|endoftext|>",
"eos_token": "<|endoftext|>",
"add_prefix_space": false
}