Files
corvus-gov-3B/tokenizer_config.json
T
ModelHub XC 1bd4836ef0 初始化项目,由ModelHub XC社区提供模型
Model: JavaFamily/corvus-gov-3B
Source: Original Platform
2026-11-11 16:36:12 +08:00

19 lines
433 B
JSON

{
"backend": "tokenizers",
"bos_token": "<|begin_of_text|>",
"clean_up_tokenization_spaces": true,
"eos_token": "<|eot_id|>",
"extra_special_tokens": {
"<|eom_id|>": "<|eom_id|>"
},
"is_local": true,
"local_files_only": false,
"model_input_names": [
"input_ids",
"attention_mask"
],
"model_max_length": 131072,
"pad_token": "<|eot_id|>",
"padding_side": "left",
"split_special_tokens": false
}