初始化项目，由ModelHub XC社区提供模型
Model: hachi-intelligence/HACHI-Summary-Ja-sarashina2.2-0.5b-instruct-v0.1 Source: Original Platform
This commit is contained in:
ModelHub XC
35
.gitattributes
vendored
Normal file
35
.gitattributes
vendored
Normal file
@@ -0,0 +1,35 @@
|
|||||||
|
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.model filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||||
|
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||||
168
README.md
Normal file
168
README.md
Normal file
@@ -0,0 +1,168 @@
|
|||||||
|
---
|
||||||
|
license: apache-2.0
|
||||||
|
base_model:
|
||||||
|
- sbintuitions/sarashina2.2-0.5b-instruct-v0.1
|
||||||
|
library_name: transformers
|
||||||
|
task_categories:
|
||||||
|
- text-generation
|
||||||
|
language:
|
||||||
|
- ja
|
||||||
|
tags:
|
||||||
|
- HACHI-Intelligence
|
||||||
|
- extractive-summarization
|
||||||
|
- SLM
|
||||||
|
datasets:
|
||||||
|
- hachi-intelligence/JapaneseSummarization-FW2EduJa-Distill
|
||||||
|
model_type: causal-lm
|
||||||
|
pipeline_tag: text-generation
|
||||||
|
finetuned_from: sbintuitions/sarashina2.2-0.5b-instruct-v0.1
|
||||||
|
---
|
||||||
|
|
||||||
|
# HACHI-Summary-Ja-sarashina2.2-0.5b-instruct-v0.1
|
||||||
|
|
||||||
|
For a comprehensive deep dive into the model's architecture, training methodology, and performance benchmarks, please follow the pawprints to our [technical report (available in Japanese)](https://zenn.dev/hachi_intelli/articles/f72205c178f133). 🐕🐾
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
**HACHI-Summary-Ja-sarashina2.2-0.5b-instruct-v0.1** is a commercially viable Japanese Small Language Model (SLM) optimized for high-performance **extractive summarization**. It is specifically designed for professional sectors where factual integrity is non-negotiable, such as **legal affairs, research, healthcare, and finance**.
|
||||||
|
|
||||||
|
Built upon SB Intuitions' **sarashina2.2-0.5b-instruct-v0.1**, this model utilizes a knowledge distillation approach to inherit advanced processing capabilities from frontier LLMs while maintaining a lightweight footprint.
|
||||||
|
|
||||||
|
### Key Features and Use Cases
|
||||||
|
* **High-Fidelity Extractive Summarization**: Expertly preserves proper nouns, numerical data, units, chronological order, and causal relationships within reports and articles.
|
||||||
|
* **Optimized for Edge Deployment**: With only 0.5B parameters, it enables rapid inference and integration into resource-constrained environments or edge devices.
|
||||||
|
* **Commercial Readiness**: Released under the Apache-2.0 license, providing a transparent and reliable AI foundation for commercial applications, modifications, and redistribution.
|
||||||
|
|
||||||
|
## Evaluation
|
||||||
|
|
||||||
|
### Benchmarking Methodology
|
||||||
|
Traditional benchmarks like **XLSUM-ja** primarily measure "abstractive summarization" (paraphrasing based on context). For models like HACHI-Summary-Ja, which prioritize the precise transcription of original data points, standard abstractive scores may not fully reflect their utility.
|
||||||
|
|
||||||
|
To address this, we developed **HES-Ja (HACHI-Extractive-Summarization-Ja)**. We extracted 100 test cases from XLSUM and created a new evaluation set focused on the exact preservation of proper nouns, numerical values (including character-specific notations), and logical flow.
|
||||||
|
|
||||||
|
> [!NOTE]
|
||||||
|
> Detailed articles and the full HES-Ja dataset are scheduled for public release in the near future.
|
||||||
|
|
||||||
|
### Benchmark Results
|
||||||
|
|
||||||
|
#### XLSUM-ja Evaluation (Abstractive Metrics)
|
||||||
|
Performance in general summarization tasks. The model shows significant strength when specific character constraints are applied.
|
||||||
|
|
||||||
|
| Model | BLEU | ROUGE-2 | ROUGE-L |
|
||||||
|
| :--- | :---: | :---: | :---: |
|
||||||
|
| Qwen3-0.6B | 0.0147 | 0.0419 | 0.0837 |
|
||||||
|
| gemma-3-270m-it | 0.0328 | **0.0788** | 0.1549 |
|
||||||
|
| granite-4.0-350m | 0.0343 | 0.0728 | 0.1575 |
|
||||||
|
| sarashina2.2-0.5b-instruct-v0.1 (Base) | 0.0263 | 0.0731 | 0.1317 |
|
||||||
|
| HACHI-Summary-Ja | 0.0276 | 0.0747 | 0.1363 |
|
||||||
|
| **HACHI-Summary-Ja (approx. 100 chars)** | **0.0412** | 0.0748 | **0.2055** |
|
||||||
|
|
||||||
|
#### HES-Ja Evaluation (Extractive Metrics)
|
||||||
|
This benchmark measures the accuracy of information transcription. **HACHI-Summary-Ja outperforms all other tested SLMs across every metric.**
|
||||||
|
|
||||||
|
| Model | BLEU | ROUGE-2 | ROUGE-L |
|
||||||
|
| :--- | :---: | :---: | :---: |
|
||||||
|
| Qwen3-0.6B | 0.0963 | 0.1496 | 0.1820 |
|
||||||
|
| gemma-3-270m-it | 0.1960 | 0.3029 | 0.3403 |
|
||||||
|
| granite-4.0-350m | 0.1003 | 0.1963 | 0.2330 |
|
||||||
|
| sarashina2.2-0.5b-instruct-v0.1 (Base) | 0.2457 | 0.3394 | 0.3635 |
|
||||||
|
| **HACHI-Summary-Ja** | **0.2757** | **0.3644** | **0.4044** |
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
```python
|
||||||
|
import torch
|
||||||
|
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
|
||||||
|
|
||||||
|
# ===== settings =====
|
||||||
|
|
||||||
|
# "","三行","100", "300", "500"から要約条件を設定
|
||||||
|
SUMMARY_MODE = ""
|
||||||
|
|
||||||
|
# 要約したいテキスト
|
||||||
|
TEXT = """忠犬ハチ公
|
||||||
|
|
||||||
|
犬種は秋田犬（あきたいぬ）で、性別はオス。名前はハチ。ハチ公の愛称でも知られる。
|
||||||
|
ハチが飼い主を待ち続けた渋谷駅の出入り口の前には、ハチの銅像が設置されており、この「忠犬ハチ公像」は、渋谷のシンボルとして、観光名所としても有名である。
|
||||||
|
|
||||||
|
ハチは、飼い主が死去した後も駅前で帰りを待ち続けた「忠犬」として知られる。東京・渋谷をはじめ、ゆかりの地には像が置かれている。
|
||||||
|
特に、渋谷駅前のハチ公銅像は、いつしか待ち合わせの目印として使われるようになり、その銅像周囲は待ち合わせ場所としては「ハチ公前」などと呼ばれ、広く親しまれている。
|
||||||
|
|
||||||
|
ハチの飼い主は、東京府豊多摩郡渋谷町大向（現・東京都渋谷区松濤一丁目）に住んでいた、東京帝国大学の教授・上野英三郎であった。
|
||||||
|
彼は、大変な愛犬家であり、ハチの前にもたくさんの犬を飼っていた。出かける時には、渋谷駅までハチを伴うことも多かった。
|
||||||
|
しかしながら、ハチを飼い始めた翌年にあたる1925年（大正14年）5月21日に上野は急死した。
|
||||||
|
|
||||||
|
上野の死後も、駅前で亡くなった飼い主の帰りを毎日待ち続けたハチの姿は、新聞記事に掲載され、人々に感銘を与えたことから「忠犬ハチ公」と呼ばれるようになった。
|
||||||
|
"""
|
||||||
|
# 出典:忠犬ハチ公(https://ja.wikipedia.org/wiki/%E5%BF%A0%E7%8A%AC%E3%83%8F%E3%83%81%E5%85%AC)
|
||||||
|
|
||||||
|
# ===== main =====
|
||||||
|
|
||||||
|
SUMMARY_PROMPT_MAP = {
|
||||||
|
"100": "原文を100字程度で簡潔に要約してください。",
|
||||||
|
"300": "原文を300字程度で簡潔に要約してください。",
|
||||||
|
"500": "原文を500字程度で簡潔に要約してください。",
|
||||||
|
"三行": "原文を三行で簡潔に要約してください。",
|
||||||
|
}
|
||||||
|
|
||||||
|
SYSTEM_PROMPT = SUMMARY_PROMPT_MAP.get(
|
||||||
|
SUMMARY_MODE,
|
||||||
|
"原文を簡潔に要約してください。"
|
||||||
|
)
|
||||||
|
|
||||||
|
model_name = "hachi-intelligence/HACHI-Summary-Ja-sarashina2.2-0.5b-instruct-v0.1"
|
||||||
|
|
||||||
|
tokenizer = AutoTokenizer.from_pretrained(
|
||||||
|
model_name,
|
||||||
|
)
|
||||||
|
|
||||||
|
model = AutoModelForCausalLM.from_pretrained(
|
||||||
|
model_name,
|
||||||
|
torch_dtype=torch.bfloat16,
|
||||||
|
device_map="auto",
|
||||||
|
)
|
||||||
|
|
||||||
|
chat_pipeline = pipeline(
|
||||||
|
"text-generation",
|
||||||
|
model=model,
|
||||||
|
tokenizer=tokenizer
|
||||||
|
)
|
||||||
|
|
||||||
|
messages = [
|
||||||
|
{"role": "system", "content": SYSTEM_PROMPT},
|
||||||
|
{"role": "user", "content": TEXT}
|
||||||
|
]
|
||||||
|
|
||||||
|
outputs = chat_pipeline(
|
||||||
|
messages,
|
||||||
|
do_sample=False,
|
||||||
|
# temperature=0.1,
|
||||||
|
repetition_penalty=1.00,
|
||||||
|
max_new_tokens=512,
|
||||||
|
num_return_sequences=1,
|
||||||
|
)
|
||||||
|
|
||||||
|
generated_messages = outputs[0]["generated_text"]
|
||||||
|
summary = generated_messages[-1]["content"]
|
||||||
|
|
||||||
|
|
||||||
|
# ===== result =====
|
||||||
|
print("\n===== SYSTEM PROMPT =====")
|
||||||
|
print(SYSTEM_PROMPT)
|
||||||
|
print("===== SOURCE TEXT =====")
|
||||||
|
print(TEXT,f"({len(TEXT)}文字)")
|
||||||
|
print("\n===== SUMMARY =====")
|
||||||
|
print(summary,f"({len(summary)}文字)")
|
||||||
|
|
||||||
|
# ===== output =====
|
||||||
|
# 秋田犬（あきたいぬ）のオス・ハチは、飼い主が死去した後も渋谷駅の出入り口で帰りを待ち続けた「忠犬ハチ公」として知られる。東京・渋谷のハチ公銅像は渋谷のシンボルとして観光名所となり、待ち合わせの目印として「ハチ公前」と呼ばれる。飼い主は東京府豊多摩郡渋谷町大向（現・東京都渋谷区松濤一丁目）の東京帝国大学教授・上野英三郎で、ハチの前に多数の犬を飼っていた。1925年（大正14年）5月21日に上野が死去した後も、ハチは渋谷駅で 飼い主の帰りを待ち続け、新聞記事で「忠犬ハチ公」と称された。 (242文字)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Acknowledgements
|
||||||
|
|
||||||
|
* [SB Intuitions (sarashina2.2-0.5b-instruct-v0.1)](https://huggingface.co/sbintuitions/sarashina2.2-0.5b-instruct-v0.1)
|
||||||
|
* [hotchpotch (fineweb-2-edu-japanese)](https://huggingface.co/datasets/hotchpotch/fineweb-2-edu-japanese)
|
||||||
|
* [Open-R1](https://github.com/huggingface/open-r1)
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
This model is licensed under the [Apache-2.0 License](https://www.google.com/search?q=LICENSE).
|
||||||
71
chat_template.jinja
Normal file
71
chat_template.jinja
Normal file
@@ -0,0 +1,71 @@
|
|||||||
|
|
||||||
|
{%- set user_messages = messages | selectattr('role', 'equalto', 'user') | list %}
|
||||||
|
{%- macro output_available_tools(tools, message) %}
|
||||||
|
{%- if tools and (message == user_messages[-1]) %}
|
||||||
|
{{- '<|available_tools|>[' }}
|
||||||
|
{%- for tool in tools %}
|
||||||
|
{%- set tool = tool.function %}
|
||||||
|
{{- "{" }}
|
||||||
|
{%- for key, val in tool.items() if key != "return" %}
|
||||||
|
{%- if val is string %}
|
||||||
|
{{- "'" + key + "': '" + val + "'" }}
|
||||||
|
{%- else %}
|
||||||
|
{{- "'" + key + "': " + val|string }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- if not loop.last %}
|
||||||
|
{{- ", " }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endfor %}
|
||||||
|
{{- "}" }}
|
||||||
|
{%- if not loop.last %}
|
||||||
|
{{- ", " }}
|
||||||
|
{%- else %}
|
||||||
|
{{- "]" }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endfor %}
|
||||||
|
{{- eos_token -}}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endmacro %}
|
||||||
|
|
||||||
|
{%- macro output_tool_results(tool_results) %}
|
||||||
|
{{- '<|tool_results|>[' }}
|
||||||
|
{%- for tool_result in tool_results %}
|
||||||
|
{{- "{'content': " + tool_result.content|string + ", 'call_id': '" + tool_result.call_id + "'}" }}
|
||||||
|
{%- endfor %}
|
||||||
|
{{- ']' }}
|
||||||
|
{{- eos_token -}}
|
||||||
|
{%- endmacro %}
|
||||||
|
|
||||||
|
{%- macro output_tool_calls(tool_calls) %}
|
||||||
|
{{- '<|tool_calls|>[' }}
|
||||||
|
{%- for tool_call in tool_calls %}
|
||||||
|
{{- "{'id': '" + tool_call.id + "', 'name': '" + tool_call.name + "', 'arguments': " + tool_call.arguments|string + '}' }}
|
||||||
|
{%- endfor %}
|
||||||
|
{{- ']' }}
|
||||||
|
{%- endmacro %}
|
||||||
|
|
||||||
|
{%- for message in messages %}
|
||||||
|
{%- if message['role'] == 'user' %}
|
||||||
|
{%- if tools is defined %}
|
||||||
|
{{- output_available_tools(tools, message) }}
|
||||||
|
{%- endif %}
|
||||||
|
{{- '<|user|>' + message['content'] + eos_token -}}
|
||||||
|
{%- elif message['role'] == 'system' %}
|
||||||
|
{{- '<|system|>' + message['content'] + eos_token -}}
|
||||||
|
{%- elif message['role'] == 'assistant' %}
|
||||||
|
{% set assistant_content = "" %}
|
||||||
|
{%- if message.content is defined %}
|
||||||
|
{% set assistant_content = message.content %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- if message.tool_calls is defined and message.tool_calls -%}
|
||||||
|
{{- '<|assistant|>' + assistant_content + output_tool_calls(message['tool_calls']) + eos_token -}}
|
||||||
|
{%- else %}
|
||||||
|
{{- '<|assistant|>' + assistant_content + eos_token }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- elif message['role'] == 'tool_results' %}
|
||||||
|
{{- output_tool_results(message.tool_results) }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- if loop.last and add_generation_prompt -%}
|
||||||
|
{{- '<|assistant|>' -}}
|
||||||
|
{%- endif -%}
|
||||||
|
{%- endfor %}
|
||||||
30
config.json
Normal file
30
config.json
Normal file
@@ -0,0 +1,30 @@
|
|||||||
|
{
|
||||||
|
"architectures": [
|
||||||
|
"LlamaForCausalLM"
|
||||||
|
],
|
||||||
|
"attention_bias": false,
|
||||||
|
"attention_dropout": 0.0,
|
||||||
|
"bos_token_id": 1,
|
||||||
|
"dtype": "bfloat16",
|
||||||
|
"eos_token_id": 2,
|
||||||
|
"head_dim": 80,
|
||||||
|
"hidden_act": "silu",
|
||||||
|
"hidden_size": 1280,
|
||||||
|
"initializer_range": 0.02,
|
||||||
|
"intermediate_size": 4480,
|
||||||
|
"max_position_embeddings": 8192,
|
||||||
|
"mlp_bias": false,
|
||||||
|
"model_type": "llama",
|
||||||
|
"num_attention_heads": 16,
|
||||||
|
"num_hidden_layers": 24,
|
||||||
|
"num_key_value_heads": 8,
|
||||||
|
"pad_token_id": 3,
|
||||||
|
"pretraining_tp": 1,
|
||||||
|
"rms_norm_eps": 1e-05,
|
||||||
|
"rope_scaling": null,
|
||||||
|
"rope_theta": 500000,
|
||||||
|
"tie_word_embeddings": false,
|
||||||
|
"transformers_version": "4.56.2",
|
||||||
|
"use_cache": false,
|
||||||
|
"vocab_size": 102400
|
||||||
|
}
|
||||||
12
generation_config.json
Normal file
12
generation_config.json
Normal file
@@ -0,0 +1,12 @@
|
|||||||
|
{
|
||||||
|
"bos_token_id": 1,
|
||||||
|
"do_sample": true,
|
||||||
|
"eos_token_id": [
|
||||||
|
2
|
||||||
|
],
|
||||||
|
"pad_token_id": 3,
|
||||||
|
"repetition_penalty": 1.05,
|
||||||
|
"temperature": 0.7,
|
||||||
|
"top_p": 0.9,
|
||||||
|
"transformers_version": "4.56.2"
|
||||||
|
}
|
||||||
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:3c360dd7dc13e16447ab5fcdfa486d9e73f86186d1b59ff16b0f1beac5d85bfa
|
||||||
|
size 1586121792
|
||||||
51
special_tokens_map.json
Normal file
51
special_tokens_map.json
Normal file
@@ -0,0 +1,51 @@
|
|||||||
|
{
|
||||||
|
"bos_token": {
|
||||||
|
"content": "<s>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
"cls_token": {
|
||||||
|
"content": "<cls>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
"eos_token": {
|
||||||
|
"content": "</s>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
"mask_token": {
|
||||||
|
"content": "<mask>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
"pad_token": {
|
||||||
|
"content": "<pad>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
"sep_token": {
|
||||||
|
"content": "<sep>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
"unk_token": {
|
||||||
|
"content": "<unk>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
}
|
||||||
|
}
|
||||||
409827
tokenizer.json
Normal file
409827
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
3
tokenizer.model
Normal file
3
tokenizer.model
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:008293028e1a9d9a1038d9b63d989a2319797dfeaa03f171093a57b33a3a8277
|
||||||
|
size 1831879
|
||||||
171
tokenizer_config.json
Normal file
171
tokenizer_config.json
Normal file
@@ -0,0 +1,171 @@
|
|||||||
|
{
|
||||||
|
"add_bos_token": false,
|
||||||
|
"add_dummy_prefix_space": false,
|
||||||
|
"add_eos_token": false,
|
||||||
|
"add_prefix_space": false,
|
||||||
|
"added_tokens_decoder": {
|
||||||
|
"0": {
|
||||||
|
"content": "<unk>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"1": {
|
||||||
|
"content": "<s>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"2": {
|
||||||
|
"content": "</s>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"3": {
|
||||||
|
"content": "<pad>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"4": {
|
||||||
|
"content": "<sep>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"5": {
|
||||||
|
"content": "<mask>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"6": {
|
||||||
|
"content": "<cls>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"7": {
|
||||||
|
"content": "<|system|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"8": {
|
||||||
|
"content": "<|assistant|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"9": {
|
||||||
|
"content": "<|user|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"10": {
|
||||||
|
"content": "<|available_tools|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"11": {
|
||||||
|
"content": "<|tool_calls|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"12": {
|
||||||
|
"content": "<|tool_results|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"13": {
|
||||||
|
"content": "<|code|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"14": {
|
||||||
|
"content": "<|file|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"102397": {
|
||||||
|
"content": "<|prefix|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"102398": {
|
||||||
|
"content": "<|suffix|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
},
|
||||||
|
"102399": {
|
||||||
|
"content": "<|middle|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": false
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"bos_token": "<s>",
|
||||||
|
"clean_up_tokenization_spaces": false,
|
||||||
|
"cls_token": "<cls>",
|
||||||
|
"do_lower_case": false,
|
||||||
|
"eos_token": "</s>",
|
||||||
|
"extra_ids": 0,
|
||||||
|
"extra_special_tokens": {},
|
||||||
|
"keep_accents": true,
|
||||||
|
"legacy": false,
|
||||||
|
"mask_token": "<mask>",
|
||||||
|
"model_max_length": 1000000000000000019884624838656,
|
||||||
|
"pad_token": "<pad>",
|
||||||
|
"padding_side": "right",
|
||||||
|
"sep_token": "<sep>",
|
||||||
|
"sp_model_kwargs": {},
|
||||||
|
"spaces_between_special_tokens": false,
|
||||||
|
"tokenizer_class": "LlamaTokenizer",
|
||||||
|
"unk_token": "<unk>",
|
||||||
|
"use_default_system_prompt": false
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user