247 lines
12 KiB
Markdown
247 lines
12 KiB
Markdown
|
---
|
||
|
|
|
||
|
|
library_name: transformers
|
||
|
|
license: other
|
||
|
|
license_name: "kanana-open-license"
|
||
|
|
license_link: https://huggingface.co/kakaocorp/kanana-2-1.3b-instruct/blob/main/LICENSE
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
model_id: kakaocorp/kanana-2-1.3b-instruct
|
||
|
|
repo: kakaocorp/kanana-2-1.3b-instruct
|
||
|
|
developers: Kanana LLM
|
||
|
|
base_model:
|
||
|
|
|
||
|
|
- kakaocorp/kanana-2-1.3b-base
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
|
||
|
|
<p align="center">
|
||
|
|
<img src="./assets/logo/kanana.png" width="60%" alt="Kanana">
|
||
|
|
</p>
|
||
|
|
|
||
|
|
<p align="center">
|
||
|
|
🤗 <a href="https://huggingface.co/collections/kakaocorp/kanana-2-slm">HF Models</a> | 📕 <a href="https://tech.kakao.com/posts/826">Blog</a>
|
||
|
|
</p>
|
||
|
|
<br><br>
|
||
|
|
|
||
|
|
## News 🔥
|
||
|
|
|
||
|
|
- `2026/07/27`: 🤗 Released `kanana-2-3b`, `kanana-2-1.3b` HF model weights.
|
||
|
|
- `2026/07/27`: 📕 Published a blog post about the development of the `Kanana-2 SLM` series.
|
||
|
|
|
||
|
|
# Introduction
|
||
|
|
|
||
|
|
We present Kanana-2 SLM, **Kakao's second series of Small Language Models (SLMs)**, designed to deliver strong language capabilities while remaining compact and efficient for practical deployment. The series includes 3B, 1.3B, and 0.9B models. This release publicly includes the 3B model and the compressed 1.3B model, providing a balance between capability and efficiency for a wide range of applications.
|
||
|
|
Kanana-2-3B was **pretrained from scratch on TPU clusters** and further improved through post-training with **supervised fine-tuning and reinforcement learning**, resulting in strong instruction-following and reasoning capabilities.
|
||
|
|
Kanana-2-1.3B models are derived from Kanana-2-3B through a **cascade pruning and distillation pipeline**. To further improve deployment efficiency, they adopt **Sliding Window Attention (SWA)**, enabling memory-efficient long-context inference with support for context lengths of up to **32K tokens** while substantially reducing KV-cache memory requirements.
|
||
|
|
|
||
|
|
The Kanana-2 SLM release consists of the following four publicly available models:
|
||
|
|
|
||
|
|
- **Kanana-2-3B-Base** — 3B pretrained base
|
||
|
|
- **Kanana-2-3B-Instruct** — instruction-tuned 3B model
|
||
|
|
- **Kanana-2-1.3B-Base** — compressed 1.3B pretrained base
|
||
|
|
- **Kanana-2-1.3B-Instruct** — instruction-tuned 1.3B model for on-device deployment
|
||
|
|
|
||
|
|
> [!NOTE]
|
||
|
|
> No Kakao user data was used for either pre-training or post-training.
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
# Highlights
|
||
|
|
|
||
|
|
- **Cascade Pruning & Distillation**: Kanana-2-1.3B is built by progressively compressing Kanana-2-3B-Base (3B → 2B → 1.3B → 0.9B) through a cascade pruning and distillation pipeline.
|
||
|
|
- **Sliding Window Attention (SWA)**: Uses a 3:1 hybrid layout of sliding-window and full-attention layers. A sliding-window size of 1024 reduces per-token KV-cache reads, cutting KV-cache usage by up to ~72.7% at a 32K context length compared to a full-attention-only model. YaRN is applied to full-attention layers, while SWA layers retain RoPE, preserving long-range context without sacrificing local-attention efficiency.
|
||
|
|
- **Kanana-2 tokenizer**: Improves Korean tokenization efficiency by over 30% compared to the previous generation.
|
||
|
|
- **Long context**: Natively supports context lengths of up to 32,768 tokens.
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
## Model Downloads
|
||
|
|
|
||
|
|
|
||
|
|
| **Model** | **Download** |
|
||
|
|
| ---------------------- | ------------------------------------------------------------------------- |
|
||
|
|
| kanana-2-3b-base | [🤗 HuggingFace](https://huggingface.co/kakaocorp/kanana-2-3b-base) |
|
||
|
|
| kanana-2-3b-instruct | [🤗 HuggingFace](https://huggingface.co/kakaocorp/kanana-2-3b-instruct) |
|
||
|
|
| kanana-2-1.3b-base | [🤗 HuggingFace](https://huggingface.co/kakaocorp/kanana-2-1.3b-base) |
|
||
|
|
| kanana-2-1.3b-instruct | [🤗 HuggingFace](https://huggingface.co/kakaocorp/kanana-2-1.3b-instruct) |
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
## Performance
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
### Base model evaluation results
|
||
|
|
|
||
|
|
|
||
|
|
<!-- prettier-ignore -->
|
||
|
|
<table>
|
||
|
|
<thead>
|
||
|
|
<tr>
|
||
|
|
<th>Benchmark</th>
|
||
|
|
<th>Metric</th>
|
||
|
|
<th>Shot</th>
|
||
|
|
<th>kanana-2-3b-base</th>
|
||
|
|
<th>kanana-2-1.3b-base</th>
|
||
|
|
<th>Qwen3-1.7B-Base</th>
|
||
|
|
</tr>
|
||
|
|
</thead>
|
||
|
|
<tbody>
|
||
|
|
<tr><td align="center" colspan="6"><b>General Tasks</b></td></tr>
|
||
|
|
<tr><td>MMLU</td><td>acc</td><td>5</td><td>62.77</td><td>56.49</td><td>62.32</td></tr>
|
||
|
|
<tr><td>MMLU-Pro</td><td>acc</td><td>5</td><td>36.54</td><td>29.87</td><td>37.08</td></tr>
|
||
|
|
<tr><td>BBH</td><td>acc</td><td>3</td><td>54.23</td><td>45.58</td><td>53.60</td></tr>
|
||
|
|
<tr><td>SimpleQA<sup>†</sup></td><td>acc</td><td>5</td><td>27.39</td><td>23.76</td><td>16.83</td></tr>
|
||
|
|
<tr><td align="center" colspan="6"><b>Mathematics Tasks</b></td></tr>
|
||
|
|
<tr><td>MATH</td><td>em</td><td>4</td><td>35.08</td><td>30.82</td><td>41.74</td></tr>
|
||
|
|
<tr><td>GSM8K</td><td>em</td><td>8</td><td>61.03</td><td>51.93</td><td>75.74</td></tr>
|
||
|
|
<tr><td align="center" colspan="6"><b>Coding Tasks</b></td></tr>
|
||
|
|
<tr><td>HumanEval</td><td>pass@1</td><td>0</td><td>55.96</td><td>51.33</td><td>45.31</td></tr>
|
||
|
|
<tr><td>MBPP</td><td>pass@1</td><td>3</td><td>50.95</td><td>44.86</td><td>53.85</td></tr>
|
||
|
|
<tr><td align="center" colspan="6"><b>Korean Tasks</b></td></tr>
|
||
|
|
<tr><td>KMMLU</td><td>acc</td><td>5</td><td>47.92</td><td>44.17</td><td>43.08</td></tr>
|
||
|
|
<tr><td>KoSimpleQA<sup>†</sup></td><td>acc</td><td>5</td><td>32.50</td><td>28.50</td><td>13.40</td></tr>
|
||
|
|
<tr><td>HAE-RAE Bench (v1.0)</td><td>acc</td><td>5</td><td>80.66</td><td>75.34</td><td>55.54</td></tr>
|
||
|
|
<tr><td>MATH-Ko<sup>‡</sup></td><td>em</td><td>4</td><td>31.54</td><td>24.75</td><td>31.92</td></tr>
|
||
|
|
<tr><td>MBPP-Ko<sup>§</sup></td><td>pass@1</td><td>3</td><td>44.55</td><td>39.38</td><td>47.97</td></tr>
|
||
|
|
<tr><td align="center" colspan="6"><b>Long Context Tasks</b></td></tr>
|
||
|
|
<tr><td>RULER-32K</td><td>acc</td><td>0</td><td>67.50</td><td>55.72</td><td>69.01</td></tr>
|
||
|
|
</tbody>
|
||
|
|
</table>
|
||
|
|
|
||
|
|
|
||
|
|
† Evaluated in Multiple Choice Question Answering (MCQA) format with 10 options.
|
||
|
|
|
||
|
|
‡ Subsets from [HRM8K](https://huggingface.co/datasets/HAERAE-HUB/HRM8K) (MATH, GSM8K).
|
||
|
|
|
||
|
|
§ Internally translated to Korean.
|
||
|
|
|
||
|
|
### Instruct model evaluation results
|
||
|
|
|
||
|
|
Instruction-following, chat, tool-calling, code, math, and knowledge benchmarks for the Kanana-2 SLM series. Scores use greedy decoding (temperature 0.0, top-p 1.0, max 4096 tokens); the metric for each benchmark is listed in the Metric column.
|
||
|
|
|
||
|
|
|
||
|
|
<!-- prettier-ignore -->
|
||
|
|
<table>
|
||
|
|
<thead>
|
||
|
|
<tr>
|
||
|
|
<th>Benchmark</th>
|
||
|
|
<th>Metric</th>
|
||
|
|
<th>kanana-2-3b-instruct</th>
|
||
|
|
<th>kanana-2-1.3b-instruct</th>
|
||
|
|
<th>Qwen3.5-2B</th>
|
||
|
|
<th>Qwen3-1.7B</th>
|
||
|
|
</tr>
|
||
|
|
</thead>
|
||
|
|
<tbody>
|
||
|
|
<tr><td align="center" colspan="6"><b>Chat</b></td></tr>
|
||
|
|
<tr><td>MT-Bench<sup>†</sup></td><td>judge</td><td>7.15</td><td>6.83</td><td>6.87</td><td>6.98</td></tr>
|
||
|
|
<tr><td>KoMT-Bench<sup>†</sup></td><td>judge</td><td>6.92</td><td>6.54</td><td>5.21</td><td>5.29</td></tr>
|
||
|
|
<tr><td align="center" colspan="6"><b>Instruction Following</b></td></tr>
|
||
|
|
<tr><td>IFBench</td><td>prompt strict</td><td>33.33</td><td>34.69</td><td>24.83</td><td>18.33</td></tr>
|
||
|
|
<tr><td>IFEval</td><td>prompt strict</td><td>80.96</td><td>77.63</td><td>66.91</td><td>68.39</td></tr>
|
||
|
|
<tr><td>IHEval</td><td>pass@1</td><td>35.96</td><td>27.06</td><td>38.52</td><td>42.09</td></tr>
|
||
|
|
<tr><td align="center" colspan="6"><b>Tool Calling</b></td></tr>
|
||
|
|
<tr><td>BFCL-v3 (Live)<sup>‡</sup></td><td>pass@1</td><td>71.94</td><td>69.64</td><td>66.96</td><td>65.48</td></tr>
|
||
|
|
<tr><td>BFCL-v3 (Multi-Turn)<sup>‡</sup></td><td>pass@1</td><td>17.12</td><td>5.50</td><td>6.27</td><td>4.38</td></tr>
|
||
|
|
<tr><td align="center" colspan="6"><b>Code Generation</b></td></tr>
|
||
|
|
<tr><td>MBPP</td><td>pass@1</td><td>70.63</td><td>69.05</td><td>55.56</td><td>62.17</td></tr>
|
||
|
|
<tr><td>MBPP+</td><td>pass@1</td><td>60.05</td><td>60.85</td><td>46.83</td><td>52.65</td></tr>
|
||
|
|
<tr><td align="center" colspan="6"><b>Mathematics</b></td></tr>
|
||
|
|
<tr><td>GSM-Plus</td><td>pass@1</td><td>61.17</td><td>58.27</td><td>61.29</td><td>63.10</td></tr>
|
||
|
|
<tr><td>MATH-500</td><td>pass@1</td><td>61.20</td><td>61.40</td><td>67.80</td><td>72.00</td></tr>
|
||
|
|
<tr><td>Minerva Math</td><td>pass@1</td><td>24.44</td><td>22.43</td><td>35.18</td><td>27.21</td></tr>
|
||
|
|
<tr><td align="center" colspan="6"><b>Reasoning & Knowledge</b></td></tr>
|
||
|
|
<tr><td>MMLU-CoT</td><td>acc</td><td>61.09</td><td>60.37</td><td>69.01</td><td>66.02</td></tr>
|
||
|
|
<tr><td>KMMLU-CoT</td><td>acc</td><td>43.32</td><td>42.79</td><td>41.75</td><td>37.84</td></tr>
|
||
|
|
<tr><td>HAERAE-Bench (v1.0)-CoT</td><td>acc</td><td>43.75</td><td>44.89</td><td>27.84</td><td>27.84</td></tr>
|
||
|
|
<tr><td>KoSimpleQA</td><td>acc</td><td>22.29</td><td>17.81</td><td>3.21</td><td>2.82</td></tr>
|
||
|
|
</tbody>
|
||
|
|
</table>
|
||
|
|
|
||
|
|
|
||
|
|
† Evaluated using `gpt-4o-2024-08-06` as the judge model.
|
||
|
|
|
||
|
|
‡ `Live` denotes the average score of 6 live benchmarks, and `Multi-Turn` the average score of 4 multi-turn benchmarks.
|
||
|
|
|
||
|
|
## Deployment
|
||
|
|
|
||
|
|
`kanana-2-1.3b-instruct` uses a custom hybrid attention architecture (`Kanana2TinyForCausalLM` — a Qwen3 backbone with a 3:1 SWA/full-attention layout and per-layer-type RoPE), shipped as remote code in the repository.
|
||
|
|
|
||
|
|
> [!NOTE]
|
||
|
|
> Because the modeling code is loaded from the repository, serving requires `transformers >= 4.57` and the `--trust-remote-code` flag. For SGLang, use `--attention-backend triton` so the hybrid sliding-window attention is handled correctly.
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
### vLLM
|
||
|
|
|
||
|
|
[vLLM](https://github.com/vllm-project/vllm) is a fast and memory-optimized engine designed for high-performance LLM inference and serving.
|
||
|
|
|
||
|
|
```shell
|
||
|
|
vllm serve kakaocorp/kanana-2-1.3b-instruct \
|
||
|
|
--tensor-parallel-size 1 \
|
||
|
|
--max-model-len 32768 \
|
||
|
|
--trust-remote-code \
|
||
|
|
--enable-auto-tool-choice \
|
||
|
|
--tool-call-parser qwen3_coder
|
||
|
|
```
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
### SGLang
|
||
|
|
|
||
|
|
[SGLang](https://github.com/sgl-project/sglang) is a high-efficiency framework for serving LLMs and VLMs, enabling easy deployment of OpenAI-compatible API servers.
|
||
|
|
|
||
|
|
For SGLang, the model is served through the stock `Qwen3ForCausalLM` path instead of the remote-code `Kanana2TinyForCausalLM` class. This requires two files shipped in the `sglang/` directory of this repository:
|
||
|
|
|
||
|
|
1. `sglang/config.json` — a Qwen3-flavored config (`architectures: ["Qwen3ForCausalLM"]`, `model_type: qwen3`, no `auto_map`) that keeps the hybrid-attention fields (`layer_types`, `sliding_window`, per-layer-type `rope_parameters`). Use it in place of the default `config.json` when serving with SGLang.
|
||
|
|
2. `sglang/qwen3.py` — a patched model definition that overrides the installed `sglang/srt/models/qwen3.py`.
|
||
|
|
|
||
|
|
```shell
|
||
|
|
python3 -m sglang.launch_server \
|
||
|
|
--model-path kakaocorp/kanana-2-1.3b-instruct \
|
||
|
|
--tp 1 \
|
||
|
|
--context-length 32768 \
|
||
|
|
--attention-backend triton \
|
||
|
|
--trust-remote-code \
|
||
|
|
--tool-call-parser qwen3_coder
|
||
|
|
```
|
||
|
|
|
||
|
|
> [!NOTE]
|
||
|
|
>
|
||
|
|
> - Recommended: `sglang==0.5.1`.
|
||
|
|
> - With the Qwen3-flavored `sglang/config.json` the model itself needs no remote code, but `--trust-remote-code` is passed so the tokenizer/config are loaded without prompting.
|
||
|
|
> - Use `triton` or `fa3` for the attention backend. **Avoid** `flashinfer` — it appears to have an issue with this model and causes significant accuracy/throughput degradation.
|
||
|
|
|
||
|
|
Kanana-2-1.3B shares the Qwen3 backbone but adds a **3:1 SWA/full hybrid attention layout** and **per-layer-type RoPE**, which SGLang's stock Qwen3 model does not handle. The patched `sglang/qwen3.py` changes the decoder layer to:
|
||
|
|
|
||
|
|
1. **Per-layer-type RoPE** — for each layer, read the entry in `config.rope_parameters` matching `config.layer_types[layer_id]` and build that layer's own rotary embedding from it: `full_attention` uses YaRN (`factor=40`, `original_max_position_embeddings=4096`) and `sliding_attention` uses default RoPE (`rope_theta=10000`). A layer type with no matching entry falls back to no RoPE.
|
||
|
|
2. **Hybrid sliding-window attention** — layers typed `sliding_attention` run `RadixAttention` with `sliding_window_size = config.sliding_window - 1` (SGLang uses an exclusive window, HF an inclusive one), while `full_attention` layers use full causal attention.
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
## License
|
||
|
|
|
||
|
|
The model weights are released under the [KananaOpenLicense](https://huggingface.co/kakaocorp/kanana-2-1.3b-instruct/blob/main/LICENSE).
|
||
|
|
|
||
|
|
## Citation
|
||
|
|
|
||
|
|
```
|
||
|
|
@misc{kanana2slm2026,
|
||
|
|
title = {Kanana-2 SLM},
|
||
|
|
author = {Kanana LLM},
|
||
|
|
year = {2026},
|
||
|
|
url = {https://huggingface.co/collections/kakaocorp/kanana-2-slm}
|
||
|
|
}
|
||
|
|
```
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
## Contact
|
||
|
|
|
||
|
|
- Kanana LLM Team Technical Support: [kanana-llm@kakaocorp.com](mailto:kanana-llm@kakaocorp.com)
|
||
|
|
- Business & Partnership Contact: [alpha.k@kakaocorp.com](mailto:alpha.k@kakaocorp.com)
|
||
|
|
|