135 lines
4.6 KiB
Markdown
135 lines
4.6 KiB
Markdown
---
|
|
base_model: tokyotech-llm/Llama-3.1-Swallow-8B-Instruct-v0.3
|
|
license:
|
|
- llama3.1
|
|
- gemma
|
|
language:
|
|
- ja
|
|
- en
|
|
pipeline_tag: text-generation
|
|
tags:
|
|
- counseling
|
|
- dialogue-system
|
|
datasets:
|
|
- UEC-InabaLab/KokoroChat
|
|
---
|
|
|
|
# 🧠 Llama-3.1-KokoroChat-High: Japanese Counseling Dialogue Model
|
|
|
|
**Llama-3.1-KokoroChat-High** is a large-scale Japanese language model fine-tuned on the **entire KokoroChat dataset**—a collection of over 6,000 psychological counseling dialogues conducted via **role-play between trained counselors**. The model is capable of generating **empathetic and context-aware responses** suitable for mental health-related conversational tasks.
|
|
|
|
---
|
|
|
|
## 💡 Overview
|
|
|
|
- ✅ Fine-tuned on **2,601 dialogues** with client feedback scores between **70 and 98**
|
|
- ✅ Data collected through **text-based role-play** by trained counselors
|
|
- ✅ Covers a wide range of topics: depression, family, school, career, relationships, and more
|
|
- ✅ Base Model: [`tokyotech-llm/Llama-3.1-Swallow-8B-Instruct-v0.3`](https://huggingface.co/tokyotech-llm/Llama-3.1-Swallow-8B-Instruct-v0.3)
|
|
|
|
---
|
|
|
|
## ⚙️ Usage Example
|
|
|
|
```python
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
model_id = "UEC-InabaLab/Llama-3.1-KokoroChat-High"
|
|
|
|
# Load tokenizer and model
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
|
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
|
|
|
|
# Set pad_token_id
|
|
if tokenizer.pad_token_id is None:
|
|
tokenizer.pad_token = "[PAD]"
|
|
tokenizer.pad_token_id = tokenizer.convert_tokens_to_ids("[PAD]")
|
|
|
|
model.config.pad_token_id = tokenizer.pad_token_id
|
|
|
|
# Build dialogue input
|
|
messages = [
|
|
{"role": "system", "content": "心理カウンセリングの会話において、対話履歴を考慮し、カウンセラーとして適切に応答してください。"},
|
|
{"role": "user", "content": "最近、気分が落ち込んでやる気が出ません。"}
|
|
]
|
|
|
|
# Tokenize with chat template
|
|
inputs = tokenizer.apply_chat_template(
|
|
messages,
|
|
add_generation_prompt=True,
|
|
return_tensors="pt"
|
|
).to(model.device)
|
|
|
|
attention_mask = inputs.ne(tokenizer.pad_token_id)
|
|
|
|
# Generate response
|
|
outputs = model.generate(
|
|
inputs,
|
|
attention_mask=attention_mask,
|
|
pad_token_id=tokenizer.pad_token_id,
|
|
max_new_tokens=256
|
|
)
|
|
|
|
# Extract only the newly generated tokens
|
|
response = outputs[0][inputs.shape[-1]:]
|
|
response_text = tokenizer.decode(response, skip_special_tokens=True)
|
|
|
|
# Print clean response
|
|
print(response_text)
|
|
```
|
|
|
|
---
|
|
|
|
## 🛠️ Fine-Tuning Details
|
|
|
|
Fine-tuning was performed using **QLoRA** with the following configuration:
|
|
|
|
- **Quantization**: 4-bit NF4 with bfloat16 computation
|
|
- **LoRA target modules**: `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`
|
|
- **LoRA parameters**:
|
|
- `r = 8`
|
|
- `lora_alpha = 16`
|
|
- `lora_dropout = 0.05`
|
|
|
|
### Dataset Split
|
|
|
|
- **Training Data**: 2,601 dialogues with feedback scores between 70 and 98
|
|
- **Train/Validation Split**: 90% train, 10% validation
|
|
|
|
### Hyperparameter Settings
|
|
|
|
- **Optimizer**: `adamw_8bit`
|
|
- **Warm-up Steps**: `100`
|
|
- **Learning Rate**: `1e-3`
|
|
- **Epochs**: `5`
|
|
- **Batch Size**: `8`
|
|
- **Validation Frequency**: every 400 steps
|
|
|
|
---
|
|
|
|
## 📄 Citation
|
|
|
|
If you use this model or dataset, please cite the following paper:
|
|
|
|
```bibtex
|
|
@inproceedings{qi2025kokorochat,
|
|
title = {KokoroChat: A Japanese Psychological Counseling Dialogue Dataset Collected via Role-Playing by Trained Counselors},
|
|
author = {Zhiyang Qi and Takumasa Kaneko and Keiko Takamizo and Mariko Ukiyo and Michimasa Inaba},
|
|
booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics},
|
|
year = {2025},
|
|
url = {https://github.com/UEC-InabaLab/KokoroChat}
|
|
}
|
|
```
|
|
---
|
|
|
|
## 🔗 Related
|
|
|
|
- 📁 **Dataset**:
|
|
- [KokoroChat on Hugging Face Datasets](https://huggingface.co/datasets/UEC-InabaLab/KokoroChat)
|
|
- [KokoroChat on GitHub (UEC-InabaLab)](https://github.com/UEC-InabaLab/KokoroChat)
|
|
- 🤖 **Model Variants**:
|
|
- [Llama-3.1-KokoroChat-Low](https://huggingface.co/UEC-InabaLab/Llama-3.1-KokoroChat-Low): fine-tuned on **3,870 dialogues** with client feedback scores **< 70**
|
|
- [Llama-3.1-KokoroChat-Full](https://huggingface.co/UEC-InabaLab/Llama-3.1-KokoroChat-Full): fine-tuned on **6,471 dialogues** with client feedback scores **≤ 98**
|
|
- 🤖 **Score Prediction Model**:
|
|
- [Llama-3.1-KokoroChat-ScorePrediction](https://huggingface.co/UEC-InabaLab/Llama-3.1-KokoroChat-ScorePrediction): fine-tuned to **predict client feedback scores** based on the full counseling dialogue history.
|
|
- 📄 **Paper**: [ACL 2025 Paper](https://aclanthology.org/2025.acl-long.608/) |