169 lines
8.1 KiB
Markdown
169 lines
8.1 KiB
Markdown
|
---
|
|||
|
|
license: apache-2.0
|
|||
|
|
base_model:
|
|||
|
|
- sbintuitions/sarashina2.2-0.5b-instruct-v0.1
|
|||
|
|
library_name: transformers
|
|||
|
|
task_categories:
|
|||
|
|
- text-generation
|
|||
|
|
language:
|
|||
|
|
- ja
|
|||
|
|
tags:
|
|||
|
|
- HACHI-Intelligence
|
|||
|
|
- extractive-summarization
|
|||
|
|
- SLM
|
|||
|
|
datasets:
|
|||
|
|
- hachi-intelligence/JapaneseSummarization-FW2EduJa-Distill
|
|||
|
|
model_type: causal-lm
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
finetuned_from: sbintuitions/sarashina2.2-0.5b-instruct-v0.1
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# HACHI-Summary-Ja-sarashina2.2-0.5b-instruct-v0.1
|
|||
|
|
|
|||
|
|
For a comprehensive deep dive into the model's architecture, training methodology, and performance benchmarks, please follow the pawprints to our [technical report (available in Japanese)](https://zenn.dev/hachi_intelli/articles/f72205c178f133). 🐕🐾
|
|||
|
|
|
|||
|
|
## Overview
|
|||
|
|
**HACHI-Summary-Ja-sarashina2.2-0.5b-instruct-v0.1** is a commercially viable Japanese Small Language Model (SLM) optimized for high-performance **extractive summarization**. It is specifically designed for professional sectors where factual integrity is non-negotiable, such as **legal affairs, research, healthcare, and finance**.
|
|||
|
|
|
|||
|
|
Built upon SB Intuitions' **sarashina2.2-0.5b-instruct-v0.1**, this model utilizes a knowledge distillation approach to inherit advanced processing capabilities from frontier LLMs while maintaining a lightweight footprint.
|
|||
|
|
|
|||
|
|
### Key Features and Use Cases
|
|||
|
|
* **High-Fidelity Extractive Summarization**: Expertly preserves proper nouns, numerical data, units, chronological order, and causal relationships within reports and articles.
|
|||
|
|
* **Optimized for Edge Deployment**: With only 0.5B parameters, it enables rapid inference and integration into resource-constrained environments or edge devices.
|
|||
|
|
* **Commercial Readiness**: Released under the Apache-2.0 license, providing a transparent and reliable AI foundation for commercial applications, modifications, and redistribution.
|
|||
|
|
|
|||
|
|
## Evaluation
|
|||
|
|
|
|||
|
|
### Benchmarking Methodology
|
|||
|
|
Traditional benchmarks like **XLSUM-ja** primarily measure "abstractive summarization" (paraphrasing based on context). For models like HACHI-Summary-Ja, which prioritize the precise transcription of original data points, standard abstractive scores may not fully reflect their utility.
|
|||
|
|
|
|||
|
|
To address this, we developed **HES-Ja (HACHI-Extractive-Summarization-Ja)**. We extracted 100 test cases from XLSUM and created a new evaluation set focused on the exact preservation of proper nouns, numerical values (including character-specific notations), and logical flow.
|
|||
|
|
|
|||
|
|
> [!NOTE]
|
|||
|
|
> Detailed articles and the full HES-Ja dataset are scheduled for public release in the near future.
|
|||
|
|
|
|||
|
|
### Benchmark Results
|
|||
|
|
|
|||
|
|
#### XLSUM-ja Evaluation (Abstractive Metrics)
|
|||
|
|
Performance in general summarization tasks. The model shows significant strength when specific character constraints are applied.
|
|||
|
|
|
|||
|
|
| Model | BLEU | ROUGE-2 | ROUGE-L |
|
|||
|
|
| :--- | :---: | :---: | :---: |
|
|||
|
|
| Qwen3-0.6B | 0.0147 | 0.0419 | 0.0837 |
|
|||
|
|
| gemma-3-270m-it | 0.0328 | **0.0788** | 0.1549 |
|
|||
|
|
| granite-4.0-350m | 0.0343 | 0.0728 | 0.1575 |
|
|||
|
|
| sarashina2.2-0.5b-instruct-v0.1 (Base) | 0.0263 | 0.0731 | 0.1317 |
|
|||
|
|
| HACHI-Summary-Ja | 0.0276 | 0.0747 | 0.1363 |
|
|||
|
|
| **HACHI-Summary-Ja (approx. 100 chars)** | **0.0412** | 0.0748 | **0.2055** |
|
|||
|
|
|
|||
|
|
#### HES-Ja Evaluation (Extractive Metrics)
|
|||
|
|
This benchmark measures the accuracy of information transcription. **HACHI-Summary-Ja outperforms all other tested SLMs across every metric.**
|
|||
|
|
|
|||
|
|
| Model | BLEU | ROUGE-2 | ROUGE-L |
|
|||
|
|
| :--- | :---: | :---: | :---: |
|
|||
|
|
| Qwen3-0.6B | 0.0963 | 0.1496 | 0.1820 |
|
|||
|
|
| gemma-3-270m-it | 0.1960 | 0.3029 | 0.3403 |
|
|||
|
|
| granite-4.0-350m | 0.1003 | 0.1963 | 0.2330 |
|
|||
|
|
| sarashina2.2-0.5b-instruct-v0.1 (Base) | 0.2457 | 0.3394 | 0.3635 |
|
|||
|
|
| **HACHI-Summary-Ja** | **0.2757** | **0.3644** | **0.4044** |
|
|||
|
|
|
|||
|
|
## Usage
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
import torch
|
|||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
|
|||
|
|
|
|||
|
|
# ===== settings =====
|
|||
|
|
|
|||
|
|
# "","三行","100", "300", "500"から要約条件を設定
|
|||
|
|
SUMMARY_MODE = ""
|
|||
|
|
|
|||
|
|
# 要約したいテキスト
|
|||
|
|
TEXT = """忠犬ハチ公
|
|||
|
|
|
|||
|
|
犬種は秋田犬（あきたいぬ）で、性別はオス。名前はハチ。ハチ公の愛称でも知られる。
|
|||
|
|
ハチが飼い主を待ち続けた渋谷駅の出入り口の前には、ハチの銅像が設置されており、この「忠犬ハチ公像」は、渋谷のシンボルとして、観光名所としても有名である。
|
|||
|
|
|
|||
|
|
ハチは、飼い主が死去した後も駅前で帰りを待ち続けた「忠犬」として知られる。東京・渋谷をはじめ、ゆかりの地には像が置かれている。
|
|||
|
|
特に、渋谷駅前のハチ公銅像は、いつしか待ち合わせの目印として使われるようになり、その銅像周囲は待ち合わせ場所としては「ハチ公前」などと呼ばれ、広く親しまれている。
|
|||
|
|
|
|||
|
|
ハチの飼い主は、東京府豊多摩郡渋谷町大向（現・東京都渋谷区松濤一丁目）に住んでいた、東京帝国大学の教授・上野英三郎であった。
|
|||
|
|
彼は、大変な愛犬家であり、ハチの前にもたくさんの犬を飼っていた。出かける時には、渋谷駅までハチを伴うことも多かった。
|
|||
|
|
しかしながら、ハチを飼い始めた翌年にあたる1925年（大正14年）5月21日に上野は急死した。
|
|||
|
|
|
|||
|
|
上野の死後も、駅前で亡くなった飼い主の帰りを毎日待ち続けたハチの姿は、新聞記事に掲載され、人々に感銘を与えたことから「忠犬ハチ公」と呼ばれるようになった。
|
|||
|
|
"""
|
|||
|
|
# 出典:忠犬ハチ公(https://ja.wikipedia.org/wiki/%E5%BF%A0%E7%8A%AC%E3%83%8F%E3%83%81%E5%85%AC)
|
|||
|
|
|
|||
|
|
# ===== main =====
|
|||
|
|
|
|||
|
|
SUMMARY_PROMPT_MAP = {
|
|||
|
|
"100": "原文を100字程度で簡潔に要約してください。",
|
|||
|
|
"300": "原文を300字程度で簡潔に要約してください。",
|
|||
|
|
"500": "原文を500字程度で簡潔に要約してください。",
|
|||
|
|
"三行": "原文を三行で簡潔に要約してください。",
|
|||
|
|
}
|
|||
|
|
|
|||
|
|
SYSTEM_PROMPT = SUMMARY_PROMPT_MAP.get(
|
|||
|
|
SUMMARY_MODE,
|
|||
|
|
"原文を簡潔に要約してください。"
|
|||
|
|
)
|
|||
|
|
|
|||
|
|
model_name = "hachi-intelligence/HACHI-Summary-Ja-sarashina2.2-0.5b-instruct-v0.1"
|
|||
|
|
|
|||
|
|
tokenizer = AutoTokenizer.from_pretrained(
|
|||
|
|
model_name,
|
|||
|
|
)
|
|||
|
|
|
|||
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|||
|
|
model_name,
|
|||
|
|
torch_dtype=torch.bfloat16,
|
|||
|
|
device_map="auto",
|
|||
|
|
)
|
|||
|
|
|
|||
|
|
chat_pipeline = pipeline(
|
|||
|
|
"text-generation",
|
|||
|
|
model=model,
|
|||
|
|
tokenizer=tokenizer
|
|||
|
|
)
|
|||
|
|
|
|||
|
|
messages = [
|
|||
|
|
{"role": "system", "content": SYSTEM_PROMPT},
|
|||
|
|
{"role": "user", "content": TEXT}
|
|||
|
|
]
|
|||
|
|
|
|||
|
|
outputs = chat_pipeline(
|
|||
|
|
messages,
|
|||
|
|
do_sample=False,
|
|||
|
|
# temperature=0.1,
|
|||
|
|
repetition_penalty=1.00,
|
|||
|
|
max_new_tokens=512,
|
|||
|
|
num_return_sequences=1,
|
|||
|
|
)
|
|||
|
|
|
|||
|
|
generated_messages = outputs[0]["generated_text"]
|
|||
|
|
summary = generated_messages[-1]["content"]
|
|||
|
|
|
|||
|
|
|
|||
|
|
# ===== result =====
|
|||
|
|
print("\n===== SYSTEM PROMPT =====")
|
|||
|
|
print(SYSTEM_PROMPT)
|
|||
|
|
print("===== SOURCE TEXT =====")
|
|||
|
|
print(TEXT,f"({len(TEXT)}文字)")
|
|||
|
|
print("\n===== SUMMARY =====")
|
|||
|
|
print(summary,f"({len(summary)}文字)")
|
|||
|
|
|
|||
|
|
# ===== output =====
|
|||
|
|
# 秋田犬（あきたいぬ）のオス・ハチは、飼い主が死去した後も渋谷駅の出入り口で帰りを待ち続けた「忠犬ハチ公」として知られる。東京・渋谷のハチ公銅像は渋谷のシンボルとして観光名所となり、待ち合わせの目印として「ハチ公前」と呼ばれる。飼い主は東京府豊多摩郡渋谷町大向（現・東京都渋谷区松濤一丁目）の東京帝国大学教授・上野英三郎で、ハチの前に多数の犬を飼っていた。1925年（大正14年）5月21日に上野が死去した後も、ハチは渋谷駅で 飼い主の帰りを待ち続け、新聞記事で「忠犬ハチ公」と称された。 (242文字)
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## Acknowledgements
|
|||
|
|
|
|||
|
|
* [SB Intuitions (sarashina2.2-0.5b-instruct-v0.1)](https://huggingface.co/sbintuitions/sarashina2.2-0.5b-instruct-v0.1)
|
|||
|
|
* [hotchpotch (fineweb-2-edu-japanese)](https://huggingface.co/datasets/hotchpotch/fineweb-2-edu-japanese)
|
|||
|
|
* [Open-R1](https://github.com/huggingface/open-r1)
|
|||
|
|
|
|||
|
|
## License
|
|||
|
|
|
|||
|
|
This model is licensed under the [Apache-2.0 License](https://www.google.com/search?q=LICENSE).
|