codegen25-7b-multi-gguf-with-dummy-tokenizer/tokenizer_config.json at fa838418c0b2bd2e5af2384a82a147e196858d0a - codegen25-7b-multi-gguf-with-dummy-tokenizer - Gitea: Git with a cup of tea

sokada/codegen25-7b-multi-gguf-with-dummy-tokenizer

Files

ModelHub XC fa838418c0 初始化项目，由ModelHub XC社区提供模型

Model: sokada/codegen25-7b-multi-gguf-with-dummy-tokenizer
Source: Original Platform

2026-05-04 09:05:48 +08:00

13 lines

344 B

JSON

Raw Blame History

 {
   "add_eos_token": false,
   "add_special_tokens": true,
   "clean_up_tokenization_spaces": true,
   "eos_token": "<|endoftext|>",
   "model_max_length": 1000000000000000019884624838656,
   "pad_token": null,
   "tokenizer_class": "CodeGen25Tokenizer",
   "auto_map": {
     "AutoTokenizer": ["tokenization_codegen25.CodeGen25Tokenizer", null]
   }
 }