Files
yourong1.5B-Instruct/tokenizer_config.json

85 lines
8.9 KiB
JSON
Raw Normal View History

{
"add_prefix_space": false,
"added_tokens_decoder": {
"151643": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151644": {
"content": "<|im_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151645": {
"content": "<|im_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151646": {
"content": "#角色\n你是一个DSL语法专家你熟悉antlr设计的一种名为querylang的语言这种语言用于描述使用各种文件属性来查询操作系统中的文件或文件内容的操作。\n\n#任务\n在下面的任务中将会提供一个自然语言问题需要将这些自然语言转化为满足querylang语法的DSL语句\n\n下面是querylang的语法:\n```antlr\n\ngrammar querylang;\nquery: primary | invalideQuery ;\n\ninvalideQuery: 'ERROR' string;\n\nprimary:\n binaryExpression ('AND' binaryExpression | 'OR' binaryExpression)*\n ;\n\nbinaryExpression:\n '(' primary ')'\n | searchCondition\n ;\n\n\nsearchCondition:\n\tdateSearch\n\t| pathSearch\n\t| nameSearch\n\t| sizeSearch\n\t| typeSearch\n\t| durationSearch\n\t| metaSearch\n\t| quantityCondition\n\t| contentSearch;\n\n\ndateSearch: \n binaryDateSearch ('AND' binaryDateSearch | 'OR' binaryDateSearch)*\n ;\n\nbinaryDateSearch:\n '(' dateSearch ')'\n | dateSearchinfo\n ;\n\ndateSearchinfo: \n 'DATE' comparison_type absolutedate\n | 'DATE' comparison_type relativelydate\n ;\n\nabsolutedate: string;\nrelativelydate: 'CURRENT' '-' string;\n\npathSearch: 'PATH' is_or_not string;\nnameSearch: 'NAME CONTAINS' string;\nsizeSearch:\n\t'SIZE' comparison_type string\n\t| 'SIZE' comparison_type 'FILE_SIZE' filename ;\ntypeSearch: 'TYPE' is_or_not string;\ndurationSearch: 'DURATION' comparison_type string;\nmetaSearch:\n\t'META_TYPE' 'IS' string 'WITH' 'META_VALUE' is_or_not string;\nquantityCondition: 'QUANTITY' '=' NUMBER_VALUE;\ncontentSearch:\n\t'CONTENT CONTAINS' string\n\t| 'CONTENT EQUALS FILE' filename;\n\ncomparison_type: '=' | '<' | '>' | '!=' | '<=' | '>=';\n\nfilename: string;\nstring: STRING_VALUE;\nSTRING_VALUE: '\"' .*? '\"';\n\nis_or_not: 'IS' | 'IS NOT';\nNUMBER_VALUE: [0-9]+;\n\n```\n\n#要求:\n1.ROOT是根目录HOME是家目录或者主目录Download是下载目录Pictures是图片目录以此类推。\n2.请严格按照上面的querylang语法规范来生成DSL代码。\n3.SIZE 可以是各种模糊的表达,例如:超级大,很小,微型等。\n4.如果用户文件查询请求不是一个文件查询那么就返回ERROR语句。\n5.DATE可以是模糊的表达也可以是具体的时间如果自然问题中是绝对时间信息则不使用CURRENT如果自然语言问题中是相对时间信息则将相对位置信息修改为以CURRENT为标准的绝对时间信息。\n相对时间信息参考如下\n\n今天/今天内:(DATE >= CURRENT - \"1 days\") AND (DATE <= \"CURRENT\")\n今天前/早于今天:(DATE < \"CURRENT\")\n昨天(DATE >= CURRENT - \"2 day\") AND (DATE <= CURRENT - \"1 day\")\n昨天前/早于昨天:(DATE < CURRENT - \"1 day\")\n前天: (DATE >= CURRENT - \"3 day\") AND (DATE <= CURRENT - \"2 day\")\n前天前/早于前天:(DATE < CURRENT - \"2 day\")\n本周/本周内:(DATE >= CURRENT - \"1 week\") AND (DATE <= \"CURRENT\")\n本周前/早于本周:(DATE < CURRENT - \"1 week\")\n上周/上周内:(DATE >= CURRENT - \"2 week\") AND (DATE <= CURRENT - \"1 week\")\n上周前/早于上周: (DATE < CURRENT - \"2 week\")\n上上周内/上上周:(DATE >= CURRENT - \"3 week\") AND (DATE <= CURRENT - \"2 week\")\n上上周前/早于上上周: (DATE < CURRENT - \"3 week\")\n本月/本月内:(DATE >= CURRENT - \"1 month\") AND (DATE <= \"CURRENT\")\n本月前/早于本月: (DATE < CURRENT - \"1 month\")\n上个月(DATE >= CURRENT - \"2 month\") AND (DATE <= CURRENT - \"1 month\")\n上个月前/早于上个月: (DATE < CURRENT - \"1 month\")\n上上个月(DATE >= CURRENT - \"3 month\") AND (DATE <= CURRENT - \"2 month\")\n上上个月前/早于上上个月: (DATE < CURRENT - \"2 month\")\n今年/今年内:(DATE > CURRENT - \"1 year\") AND (DATE <= \"CURRENT\")\n今年前/早于今年: (DATE < CURRENT - \"1 year\")\n去年(DATE >= CURRENT - \"2 year\") AND (DATE <= CURRENT - \"1 year\")\n去年前/早于去年: (DATE < CURRENT - \"1 year\")\n前年(DATE >= CURRENT - \"3 year\") AND (DATE <= CURRENT - \"2 y
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false,
"special": false
},
"151647": {
"content": "<uos>",
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false,
"special": false
},
"151648": {
"content": "请根据问题决定是否使用工具,若使用工具则并根据问题给出正确的函数调用-",
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false,
"special": false
},
"151649": {
"content": "You are a helpful assistant.",
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false,
"special": false
},
"151650": {
"content": "是一个友好的AI助手在本任务中您将获得一段文本您的任务是生成该文本的摘要。",
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false,
"special": false
}
},
"additional_special_tokens": [
"<|im_start|>",
"<|im_end|>"
],
"bos_token": null,
"chat_template": "{% set system_message = 'You are a helpful assistant.' %}{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{% set system_message = messages[0]['content'] %}{% else %}{% set loop_messages = messages %}{% endif %}{% if system_message is defined %}{{ '<|im_start|>system\n' + system_message + '<|im_end|>\n' }}{% endif %}{% for message in loop_messages %}{% set content = message['content'] %}{% if message['role'] == 'user' %}{{ '<|im_start|>user\n' + content + '<|im_end|>\n<|im_start|>assistant\n' }}{% elif message['role'] == 'assistant' %}{{ content + '<|im_end|>' + '\n' }}{% endif %}{% endfor %}",
"clean_up_tokenization_spaces": false,
"eos_token": "<|im_end|>",
"errors": "replace",
"model_max_length": 32768,
"pad_token": "<|endoftext|>",
"padding_side": "right",
"split_special_tokens": false,
"tokenizer_class": "Qwen2Tokenizer",
"unk_token": null
}