192 lines
8.0 KiB
Markdown
192 lines
8.0 KiB
Markdown
---
|
||
library_name: transformers
|
||
license: apache-2.0
|
||
---
|
||
<!-- <p align="center">
|
||
<img src="https://github.com/MLP-Lab/KORMo-tutorial/blob/main/tutorial/attachment/kormo_logo.png?raw=true" style="width: 100%; max-width: 1100px;">
|
||
</p> -->
|
||
|
||
<p align="center">
|
||
<img src="https://github.com/MLP-Lab/KORMo-tutorial/blob/main/tutorial/attachment/kormo_logo.svg?raw=true" style="width: 40%; max-width: 1100px;">
|
||
</p>
|
||
|
||
|
||
|
||
|
||
## 🚀 Update News
|
||
- **2025-10-13**: Official release of KORMo-10B-base (Be aware that it's not an SFT model!!).
|
||
---
|
||
## 💡 About KORMo
|
||
**KORMo-10B** is a **10.8B parameter fully open LLM** capable of handling both **Korean and English**.
|
||
The model, training code, and training data are all **fully open**, allowing anyone to reproduce and extend them.
|
||
|
||
- **Model Size**: 10.8B parameters
|
||
- **Languages**: Korean / English
|
||
- **Training Data**: Synthetic data + public datasets (approximately 3T tokens)
|
||
- **License**: Apache 2.0
|
||
|
||
```md
|
||
The First Fully Open-Source LLM from a Non-English Region
|
||
|
||
KORMo was created with a public-interest mission: to make world-class language models accessible to everyone.
|
||
Our goal is to empower anyone to build and advance their own large language models at a global standard.
|
||
|
||
Key Features:
|
||
|
||
1. A 10B-parameter Korean–English reasoning model trained entirely from scratch.
|
||
2. 100% open resources — including all training data, code, intermediate checkpoints, and tutorials — allowing anyone to reproduce and extend a near-SOTA model on their own.
|
||
3. 3 trillion tokens of training data released publicly, featuring never-before-shared, high-quality full-cycle Korean datasets (for pretraining, post-training, general, reasoning, and reinforcement learning).
|
||
4. A collaborative effort by eight master’s students at the KAIST Graduate School of Culture Technology (MLP Lab), documented in a 45-page research paper.
|
||
|
||
If you’ve ever used a Korean language model that performs well on benchmarks but feels strange in real use, or if fine-tuning only made it worse, you’re not alone.
|
||
|
||
KORMo solves these problems head-on.
|
||
By releasing every intermediate model and post-training dataset, we give users the freedom to build on the base model with their own data, customizing and fine-tuning it in any direction they want.
|
||
|
||
👉 "If you want a great Korean language model, now you can build it yourself. It even works with free Colab GPUs!" 🤗
|
||
```
|
||
|
||
---
|
||
|
||
## 🔗 Links
|
||
|
||
- 📖 **Technical Report**: [👉 Arxive](https://arxiv.org/pdf/2510.09426)
|
||
- 🤗 **Hugging Face**: [👉 Model Download](https://huggingface.co/KORMo-Team)
|
||
- 💻 **GitHub Repository**: [👉 Training and Inference Code](https://github.com/MLP-Lab/KORMo-tutorial)
|
||
- 🔉 **Tutorial**: [👉 Instruction Tuning over google colab](https://colab.research.google.com/github/MLP-Lab/KORMo-tutorial/blob/main/tutorial/02.sft_qlora.ipynb) [👉 Youtube Tutorial](https://www.youtube.com/@MLPLab)
|
||
|
||
---
|
||
|
||
|
||
## 📈 Benchmark Performance
|
||
|
||
### 📊 Quantitative Evaluation
|
||
|
||
| Benchmark | **KORMo-10B** | smolLM3-3B | olmo2-7B | olmo2-13B | kanana1.5-8B | qwen3-8B | llama3.1-8B | gemma3-4B | gemma3-12B |
|
||
|:-----------|---------------:|-----------:|---------:|---------:|------------:|--------:|-----------:|---------:|----------:|
|
||
| **🇺🇸 English Benchmarks** |||||||||||
|
||
| arc_challenge | 58.96 | 55.55 | 59.13 | 61.01 | 56.48 | 63.82 | 54.61 | 53.58 | 63.82 |
|
||
| arc_easy | 85.48 | 83.21 | 85.06 | 86.57 | 82.74 | 87.50 | 84.01 | 82.83 | 87.37 |
|
||
| boolq | 83.46 | 82.17 | 84.50 | 86.48 | 84.53 | 87.71 | 81.87 | 80.70 | 86.61 |
|
||
| copa | 93.00 | 91.00 | 92.00 | 93.00 | 88.00 | 92.00 | 93.00 | 89.00 | 95.00 |
|
||
| gpqa_main | 30.13 | 26.79 | 26.34 | 29.24 | 29.24 | 30.13 | 23.44 | 30.13 | 35.71 |
|
||
| hellaswag | 60.25 | 56.78 | 61.52 | 65.02 | 59.93 | 59.54 | 60.96 | 57.56 | 63.67 |
|
||
| mmlu | 67.96 | 61.37 | 62.81 | 66.85 | 63.73 | 76.95 | 65.03 | 59.60 | 73.58 |
|
||
| mmlu_global | 63.44 | 57.52 | 59.88 | 63.99 | 60.21 | 75.05 | 61.30 | 57.23 | 70.23 |
|
||
| mmlu_pro | 40.18 | 34.94 | 27.29 | 32.50 | 34.93 | 56.58 | 36.23 | 27.79 | 37.07 |
|
||
| mmlu_redux | 69.00 | 62.95 | 63.53 | 68.37 | 65.88 | 78.19 | 65.86 | 60.86 | 75.25 |
|
||
| openbookqa | 39.00 | 36.40 | 39.00 | 39.60 | 36.80 | 39.20 | 39.00 | 37.00 | 40.20 |
|
||
| piqa | 81.12 | 78.45 | 80.79 | 82.64 | 80.30 | 79.05 | 80.90 | 79.49 | 82.59 |
|
||
| social_iqa | 52.81 | 50.72 | 55.89 | 57.57 | 57.01 | 56.96 | 53.12 | 51.84 | 56.45 |
|
||
| **English Avg.** | **63.45** | 59.83 | 61.36 | 64.06 | 61.52 | 67.90 | 61.49 | 59.05 | 66.73 |
|
||
| **🇰🇷 Korean Benchmarks** |||||||||||
|
||
| click | 55.29 | 46.97 | 37.79 | 41.80 | 62.76 | 60.70 | 49.22 | 49.62 | 62.21 |
|
||
| csatqa | 38.00 | 26.67 | 19.33 | 24.67 | 44.67 | 52.00 | 28.67 | 28.67 | 31.33 |
|
||
| haerae | 68.29 | 55.82 | 31.62 | 37.58 | 80.75 | 67.19 | 53.25 | 60.68 | 74.34 |
|
||
| k2_eval | 84.89 | 75.23 | 49.54 | 63.43 | 84.72 | 84.72 | 76.62 | 76.39 | 85.42 |
|
||
| kobest | 75.05 | 69.13 | 57.27 | 59.02 | 81.93 | 80.05 | 70.55 | 69.33 | 77.70 |
|
||
| kobalt | 22.86 | 15.86 | 11.43 | 13.14 | 26.29 | 26.57 | 17.43 | 15.57 | 23.86 |
|
||
| kmmlu | 46.48 | 38.52 | 33.05 | 31.24 | 48.86 | 56.93 | 40.75 | 39.84 | 51.60 |
|
||
| mmlu_global (ko) | 55.16 | 44.15 | 34.00 | 36.95 | 52.65 | 61.95 | 46.34 | 46.33 | 59.68 |
|
||
| kr_clinical_qa | 77.32 | 53.97 | 48.33 | 46.22 | 65.84 | 80.00 | 63.54 | 60.00 | 77.22 |
|
||
| **Korean Avg.** | **58.15** | 47.37 | 35.82 | 39.34 | 60.94 | 63.35 | 49.60 | 49.60 | 60.37 |
|
||
|
||
|
||
### 📝 Qualitative Evaluation (LLM-as-a-Judge)
|
||
|
||
| Benchmark | KORMo-10B | smolLM3-3B | olmo2-7B | olmo2-13B | kanana1.5-8B | qwen3-8B | llama3.1-8B | exaone3.5-8B | gemma3-12B |
|
||
|:----------|---------:|----------:|---------:|---------:|------------:|--------:|------------:|-------------:|-----------:|
|
||
| MT-Bench (EN) | 8.32 | 7.15 | 7.32 | 7.64 | 8.45 | 8.70 | 6.32 | 8.15 | 8.70 |
|
||
| KO-MT-Bench (KO) | 8.54 | - | - | - | 8.02 | 8.16 | 4.27 | 8.13 | 8.51 |
|
||
| LogicKor (KO) | 8.96 | - | - | - | 8.94 | 8.63 | 6.45 | 9.20 | 8.46 |
|
||
| **Average** | **8.61** | - | - | - | **8.47** | **8.50** | **5.68** | **8.49** | **8.56** |
|
||
|
||
---
|
||
|
||
## 📦 Installation
|
||
|
||
```bash
|
||
git clone https://github.com/MLP-Lab/KORMo-tutorial.git
|
||
cd KORMo-tutorial
|
||
bash setup/create_uv_venv.sh
|
||
source .venv_kormo/bin/activate
|
||
```
|
||
|
||
---
|
||
## 🚀 Inference Example
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
import torch
|
||
|
||
model_name = "KORMo-Team/KORMo-10B-sft"
|
||
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
||
model = AutoModelForCausalLM.from_pretrained(
|
||
model_name,
|
||
torch_dtype=torch.bfloat16,
|
||
device_map="auto",
|
||
trust_remote_code=True
|
||
)
|
||
|
||
messages = [
|
||
{"role": "user", "content": "What happens inside a black hole?"}
|
||
]
|
||
|
||
chat_prompt = tokenizer.apply_chat_template(
|
||
messages,
|
||
tokenize=False,
|
||
add_generation_prompt=True,
|
||
enable_thinking=False
|
||
)
|
||
|
||
inputs = tokenizer(chat_prompt, return_tensors="pt").to(model.device)
|
||
|
||
with torch.inference_mode():
|
||
output_ids = model.generate(
|
||
**inputs,
|
||
max_new_tokens=1024,
|
||
)
|
||
|
||
response = tokenizer.decode(output_ids[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True)
|
||
print("Assistant:", response)
|
||
```
|
||
|
||
## 🧠 Enabling Thinking Mode
|
||
|
||
If you want to enable the **thinking** mode, simply set `enable_thinking=True`:
|
||
|
||
```python
|
||
chat_prompt = tokenizer.apply_chat_template(
|
||
messages,
|
||
tokenize=False,
|
||
add_generation_prompt=True,
|
||
enable_thinking=True
|
||
)
|
||
```
|
||
---
|
||
|
||
|
||
|
||
## Contact
|
||
- KyungTae Lim, Professor at KAIST. `ktlim@kaist.ac.kr`
|
||
|
||
|
||
|
||
## Acknowledgments
|
||
- This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (RS-2025-02653113, High-Performance Research AI Computing Infrastructure Support at the 2 PFLOPS Scale)
|
||
|
||
|
||
## Citation
|
||
|
||
```text
|
||
@misc{KORMo,
|
||
author = {Minjun Kim, Hyeonseok Lim, Hangyeol Yoo, Inho Won, Seungwoo Song, Minkyung Cho, Junghun Yuk, Changsu Choi, Dongjae Shin, Huije Lee, Hoyun Song, Alice Oh and KyungTae Lim},
|
||
title = {KORMo: Korean Open Reasoning Model for Everyone},
|
||
year = {2025},
|
||
publisher = {GitHub},
|
||
journal = {Technical Report},
|
||
paperLink = {\url{https://arxiv.org/abs/2510.09426}},
|
||
},
|
||
}
|
||
```
|