初始化项目，由ModelHub XC社区提供模型
Model: KOREAson/KO-REAson-G3-4B-0831 Source: Original Platform
This commit is contained in:
ModelHub XC
242
README.md
Normal file
242
README.md
Normal file
@@ -0,0 +1,242 @@
|
||||
---
|
||||
library_name: transformers
|
||||
tags: []
|
||||
---
|
||||
|
||||
# KO-REAson
|
||||
|
||||
**KO-REAson** is a series of Korean-centric reasoning language models developed in collaboration with [OneLineAI](https://onelineai.com/), [KISTI-KONI](https://huggingface.co/KISTI-KONI), [HAE-RAE](https://huggingface.co/HAERAE-HUB) and ORACLE.
|
||||
|
||||
We use the **Language-Mixed Chain-of-Thought (CoT)** approach, which allows the model to alternate between English and Korean during the “Think” stage of reasoning, preserving key Korean terms while leveraging English for logical scaffolding.
|
||||
|
||||
Top-performing models of our series [KO-REAson-AX3_1-7B-0831 (KONI-7B-R-20250831)](https://huggingface.co/KISTI-KONI/KONI-7B-R-20250831) and [KO-REAson-7B-Q2_5-0831](https://huggingface.co/KoReason/KO-REASon-7B-Q2_5-0831) show performance comparable to models trained on closed-source datasets such as Exaone-Deep-7.8B.
|
||||
|
||||
|
||||
<p align="left">
|
||||
<img src="https://cdn-uploads.huggingface.co/production/uploads/60d3e619b8448e1785bbda2a/uqrKdxbQEqAFknYBmuH7Y.png"
|
||||
alt="Model Comparison" width="750"/>
|
||||
<br>
|
||||
<em style="display:inline-block; max-width:750px; text-align:cener; white-space:normal; word-wrap:break-word; line-height:1.5;">
|
||||
<b>Left:</b> Average performance (Held-out-Ko) of open models trained on closed or open data;
|
||||
our models are highlighted in green.
|
||||
</em>
|
||||
</p>
|
||||
|
||||
## Model Details
|
||||
|
||||
The **KO-REAson-0831** family comes in six variants based on the base model used.
|
||||
|
||||
| Model (link) | Base | Notes |
|
||||
| -------------------------------------------------------------------------------------------- | -------------------- | --------------------------- |
|
||||
| [KO-REAson-L3_1-8B-0831](https://huggingface.co/KoReason/KO-REASon-L3_1-8B-0831) | [Llama-3.1-8B](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) | `L3_1` → Llama-3.1-8B |
|
||||
| [KO-REAson-KL3_1-8B-0831](https://huggingface.co/KOREAson/KO-REAson-KL3_1-8B-0831) | [Koni-Llama-3.1-8B](https://huggingface.co/KISTI-KONI/KONI-Llama3.1-8B-Instruct-20241024) | `KL3_1` → Koni-Llama-3.1-8B; also called [KONI-Llama3.1-8B-R-20250831](https://huggingface.co/KISTI-KONI/KONI-Llama3.1-8B-R-20250831) |
|
||||
| [KO-REAson-G3-4B-0831](https://huggingface.co/KoReason/KO-REASon-G3-4B-0831) | [Gemma-3 4B](https://huggingface.co/google/gemma-3-4b-it) | `G3` → Gemma-3-4B |
|
||||
| [KO-REAson-AX3_1-7B-0831](https://huggingface.co/KOREAson/KO-REAson-7B-AX3_1-0831) | [A.X.-3.1-Light (≈7B)](https://huggingface.co/skt/A.X-3.1-Light) | `AX3_1` → A.X.-3.1-Light; also called [KONI-7B-R-20250831](https://huggingface.co/KISTI-KONI/KONI-7B-R-20250831) |
|
||||
| [KO-REAson-K2505_8B-0831](https://huggingface.co/KoReason/KO-REASon-K2505_8B-0831) | [Kanana-2505 (8B)](https://huggingface.co/kakaocorp/kanana-1.5-8b-instruct-2505) | `K2505` → Kanana-2505 |
|
||||
| [KO-REAson-7B-Q2_5-0831](https://huggingface.co/KoReason/KO-REASon-7B-Q2_5-0831) | [Qwen-2.5 (7B)](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) | `Q2_5` → Qwen-2.5 |
|
||||
|
||||
|
||||
|
||||
# Performance
|
||||
|
||||
**Evaluation Datasets**
|
||||
|
||||
The model's performance was evaluated across a total of 11 benchmarks, and the evaluation suite is divided into two parts: (You can check these benchmarks in [HAERAE-HUB/KoSimpleEval](https://huggingface.co/datasets/HAERAE-HUB/KoSimpleEval))
|
||||
|
||||
- **Held-in**: This set of benchmarks is used for routine monitoring of the model's performance during the training and ablation study phases.
|
||||
- **Held-out**: This set is used only once to evaluate the final model after all training and ablations are complete.
|
||||
|
||||
This separation is designed to prevent inadvertent overfitting to the benchmarks during the iterative training process and to provide a more accurate measure of the model's generalization capabilities.
|
||||
|
||||
|**Category**|**Held-in**|**Held-out**|
|
||||
|---|---|---|
|
||||
|**General Knowledge**|KMMLU-Redux|KMMLU-HARD, KMMLU-Pro|
|
||||
|**Reasoning**|MCLM|KSM, GPQA, AIME2024, AIME2025|
|
||||
|**Korean-specific**|HAE-RAE Bench|CLIcK, KoBALT-700|
|
||||
|
||||
|
||||
**Comparison with models trained on public datasets**
|
||||
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Models</th>
|
||||
<th># Instances</th>
|
||||
<th>Methodology</th>
|
||||
<th>Held-Out (Ko)</th>
|
||||
<th>Held-Out (En)</th>
|
||||
<th>Total</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<th>KO-REASon-AX3_1-7B-0831(KONI-7B-R-20250831; Ours)</th>
|
||||
<td>260k</td>
|
||||
<td>SFT</td>
|
||||
<td><b>44.6</b></td>
|
||||
<td>41.2</td>
|
||||
<td><u>43.3</u></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>KO-REASon-7B-Q2_5-0831(Ours)</th>
|
||||
<td>260k</td>
|
||||
<td>SFT</td>
|
||||
<td><b>45.10</b></td>
|
||||
<td>38.75</td>
|
||||
<td><u>49.95</u></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>KO-REAson-KL3_1-8B-0831(KONI-Llama3.1-8B-R-20250831)</th>
|
||||
<td>260k</td>
|
||||
<td>SFT</td>
|
||||
<td>40.13</td>
|
||||
<td>30.57</td>
|
||||
<td>43.66</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td colspan="6" style="text-align:center; font-weight:bold;">Open Recipe (En)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>OpenThinker3-7B</th>
|
||||
<td>1.2M</td>
|
||||
<td>SFT</td>
|
||||
<td>33.6</td>
|
||||
<td><b>55.5</b></td>
|
||||
<td>41.8</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>s1.1-7B</th>
|
||||
<td>1k</td>
|
||||
<td>SFT</td>
|
||||
<td>35.6</td>
|
||||
<td>23.4</td>
|
||||
<td>31.1</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>Llama-3.1-Nemotron-Nano-8B-v1</th>
|
||||
<td>>3M</td>
|
||||
<td>SFT & RL</td>
|
||||
<td>27.0</td>
|
||||
<td>44.1</td>
|
||||
<td>33.4</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td colspan="6" style="text-align:center; font-weight:bold;">Open Recipe (Ko)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>Ko-R1-14B</th>
|
||||
<td>45k</td>
|
||||
<td>SFT</td>
|
||||
<td><u>43.7</u></td>
|
||||
<td><u>46.3</u></td>
|
||||
<td><b>44.7</b></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>Ko-R1-7B</th>
|
||||
<td>45k</td>
|
||||
<td>SFT</td>
|
||||
<td>27.3</td>
|
||||
<td>36.1</td>
|
||||
<td>30.6</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>LLaMa-3.1-Ko-Reasoning-8B</th>
|
||||
<td>63k</td>
|
||||
<td>SFT</td>
|
||||
<td>17.7</td>
|
||||
<td>7.7</td>
|
||||
<td>14.0</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Held-out benchmark performance**
|
||||
|
||||
<table border="1" cellspacing="0" cellpadding="6">
|
||||
<thead>
|
||||
<tr>
|
||||
<th rowspan="2">Model</th>
|
||||
<th rowspan="2">Model Size</th>
|
||||
<th colspan="2">General</th>
|
||||
<th colspan="4">Reasoning</th>
|
||||
<th colspan="2">Korean-Specific</th>
|
||||
<th rowspan="2">Average<br>(Held-out)</th>
|
||||
<th rowspan="2">Average<br>(Held-out-Ko)</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>KMMLU-HARD</th>
|
||||
<th>KMMLU-Pro</th>
|
||||
<th>KSM</th>
|
||||
<th>AIME 2024</th>
|
||||
<th>AIME 2025</th>
|
||||
<th>GPQA</th>
|
||||
<th>CLIcK</th>
|
||||
<th>KoBALT-700</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td><b>Llama-3.1-Nemotron-Nano-8B</b></td>
|
||||
<td>8.03</td><td>21.47</td><td>22.89</td><td>47.06</td><td>56.67</td><td>43.33</td><td>32.32</td><td>34.54</td><td>9.29</td><td>33.45</td><td>27.05</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><b>Llama-3.1-Korean-Reasoning-8B-Instruct</b></td>
|
||||
<td>8.03</td><td>14.91</td><td>21.72</td><td>6.09</td><td>0.00</td><td>0.00</td><td>23.23</td><td>39.65</td><td>6.14</td><td>13.97</td><td>17.70</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><b>EXAONE-Deep-7.8B</b></td>
|
||||
<td>7.82</td><td><u>40.96</u></td><td>37.35</td><td><b>70.80</b></td><td><b>70.00</b></td><td><b>63.33</b></td><td><b>64.65</b></td><td>54.24</td><td>18.86</td><td><b>52.52</b></td><td>44.44</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><b>DeepSeek-R1-Distill-Qwen-7B</b></td>
|
||||
<td>7.62</td><td>0.00</td><td>23.00</td><td>56.09</td><td>60.00</td><td>40.00</td><td>43.43</td><td>0.00</td><td>8.29</td><td>28.85</td><td>17.48</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><b>DeepSeek-R1-Distill-Llama-8B</b></td>
|
||||
<td>8.03</td><td>23.22</td><td>26.26</td><td>29.97</td><td>33.33</td><td>20.00</td><td><U>46.46</u></td><td>39.05</td><td>13.29</td><td>28.95</td><td>26.36</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><b>s1.1-7B</b></td>
|
||||
<td>7.62</td><td>31.16</td><td><u>37.70</u></td><td>30.60</td><td>16.67</td><td>23.33</td><td>30.30</td><td><u>56.84</u></td><td><u>21.86</u></td><td>31.06</td><td>35.63</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><b>OpenThinker3-7B</b></td>
|
||||
<td>7.62</td><td>30.31</td><td>26.26</td><td><u>63.59</u></td><td><u>66.67</u></td><td><u>53.33</u></td><td><u>46.46</u></td><td>47.69</td><td>10.14</td><td>35.63</td><td>30.60</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><b>Ko-R1-7B</b></td>
|
||||
<td>7.61</td><td>28.46</td><td>19.31</td><td>51.61</td><td>46.67</td><td>33.33</td><td>28.28</td><td>32.48</td><td>4.71</td><td>30.61</td><td>27.31</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><b>KO-REAson-KL3_1-8B-0831(KONI-Llama3.1-8B-R-20250831)</b></td>
|
||||
<td>8.03</td><td>44.64</td><td>40.08</td><td>37.96</td><td>23.33</td><td>30.00</td><td>38.38</td><td>56.39</td><td>21.57</td><td>30.57</td><td>40.13</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><b>KO-REASon-AX3_1-7B-0831 (KONI-7B-R-20250831)</b></td>
|
||||
<td>7.26</td><td>45.57</td><td>38.13</td><td>52.80</td><td>53.33</td><td>33.33</td><td>36.87</td><td><b>62.86</b></td><td>23.43</td><td><u>43.29</u></td><td><u>44.56</u></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><b>KO-REASon-7B-Q2_5-0831</b></td>
|
||||
<td>7.26</td><td><b>46.81</b></td><td><b>44.93</b></td><td>48.11</td><td>43.33</td><td>30.00</td><td>42.93</td><td>60.65</td><td><b>25.00</b></td><td>42.72</td><td><b>45.10</b></td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
|
||||
## Citation
|
||||
```
|
||||
The paper will be released soon!
|
||||
```
|
||||
|
||||
|
||||
## Contact
|
||||
|
||||
For any questions contact us via the following email :)
|
||||
|
||||
```
|
||||
spthsrbwls123@yonsei.ac.kr
|
||||
```
|
||||
|
||||
|
||||
## Acknowlegments
|
||||
This research was supported by the Korea Institute of Science and Technology Information (KISTI) (No.(KISTI) K25L1M1C1), aimed at developing KONI (KISTI Open Neural Intelligence), a large language model specialized in science and technology.
|
||||
Reference in New Issue
Block a user