Jumini-Ko-1.2B/README.md

---
language:
- ko
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
tags:
- korean
- causal-lm
- decoder-only
- from-scratch
- instruction-tuned
- 1.2b
model-index:
- name: Jumini-Ko-1.2B
  results:
  - task: {type: text-generation, name: Korean Knowledge (HAE-RAE Bench)}
    dataset: {type: HAERAE-HUB/HAE_RAE_BENCH_1.0, name: HAE-RAE Bench}
    metrics:
    - {type: acc, name: accuracy (5-shot), value: 21.9}
  - task: {type: text-generation, name: Korean Reading (Belebele-Ko)}
    dataset: {type: facebook/belebele, name: Belebele (kor_Hang)}
    metrics:
    - {type: acc, name: accuracy (5-shot), value: 27.9}
  - task: {type: text-generation, name: KMMLU}
    dataset: {type: HAERAE-HUB/KMMLU, name: KMMLU}
    metrics:
    - {type: acc, name: accuracy (5-shot), value: 24.3}
  - task: {type: text-generation, name: KoBEST}
    dataset: {type: skt/kobest_v1, name: KoBEST}
    metrics:
    - {type: acc, name: accuracy (5-shot), value: 49.5}
---

# Jumini-Ko-1.2B

**Jumini-Ko-1.2B** is a 1.26B-parameter Korean decoder-only language model **trained from
scratch** — its architecture, tokenizer, data pipeline, and training loop were all built
in-house, and it is *not* a fine-tune of any existing model. It is a compact,
Korean-specialized model designed to run on commodity hardware.

> Among the evaluated **open non-flagship Korean baselines** (`polyglot-ko-1.3b`, `Tri-1.9B`),
> Jumini-Ko-1.2B is the **strongest on Korean knowledge (HAE-RAE) and reading comprehension
> (Belebele-Ko)** — despite being the **smallest** model compared. The flagship
> `EXAONE-4.0-1.2B`, trained on far more data/compute, is stronger on all four benchmarks.

## Highlights

- 🇰🇷 **Korean-specialized, from scratch** — Llama-3-style architecture (RoPE, GQA, SwiGLU,
  RMSNorm), 128K byte-level BPE tokenizer, trained from random initialization.
- 🥇 **Beats the size-matched `polyglot-ko-1.3b` and the larger `Tri-1.9B`** on HAE-RAE and
  Belebele-Ko (5-shot), the two Korean-language benchmarks emphasized here. (It trails
  `polyglot-ko-1.3b` on KoBEST commonsense and KMMLU, and the flagship `EXAONE-4.0-1.2B` overall.)
- 🔬 **A data-centric recipe** — we show that *which* corpus you continue-pretrain on decides
  *which* capability improves (web → commonsense, Wikipedia → knowledge).
- 📦 **Edge-friendly** — 1.26B parameters; runs comfortably on a single consumer GPU.

## Benchmark Results

Korean benchmarks via the EleutherAI `lm-evaluation-harness`, 5-shot, accuracy (%). All models
evaluated under identical settings. **Bold** = best, <u>underline</u> = second best.

| Benchmark | **Jumini-Ko-1.2B** (1.26B) | polyglot-ko-1.3b (1.43B) | Tri-1.9B (1.9B) | EXAONE-4.0-1.2B† (1.28B) |
|---|:--:|:--:|:--:|:--:|
| HAE-RAE (Korean knowledge) | <u>21.9</u> | 18.7 | 18.9 | **30.0** |
| Belebele-Ko (reading) | <u>27.9</u> | 22.4 | 22.9 | **44.7** |
| KMMLU (knowledge) | 24.3 | <u>27.8</u> | 16.6 | **32.6** |
| KoBEST (commonsense) | 49.5 | **55.9** | 50.1 | <u>50.6</u> |

<sub>† EXAONE-4.0-1.2B is a strong flagship model trained on vastly more data/compute, shown as
an aspirational reference. Against the **open same-tier** baselines (polyglot-ko-1.3b, Tri-1.9B),
Jumini leads on the Korean-specific HAE-RAE and Belebele-Ko while being the smallest model.</sub>

Jumini also beats `polyglot-ko-1.3b` on **4 of 5 HAE-RAE subtasks** (history, loan-word,
rare-word, standard-nomenclature). It trails `polyglot-ko-1.3b` on commonsense (KoBEST) and broad
knowledge (KMMLU). Full per-subtask numbers are in the technical report.

## Quickstart

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "properly59/Jumini-Ko-1.2B"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.float16, device_map="auto")

prompt = "### 질문:\n대한민국의 수도는 어디인가요?\n\n### 답변:\n"
ids = tok(tok.bos_token + prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
out = model.generate(**ids, max_new_tokens=128, do_sample=True, temperature=0.8,
                     min_p=0.05, repetition_penalty=1.2, no_repeat_ngram_size=3,
                     pad_token_id=tok.pad_token_id)
print(tok.decode(out[0][ids.input_ids.shape[1]:], skip_special_tokens=True))
```

## Model Details

| | |
|---|---|
| Architecture | Decoder-only Transformer (Llama-3 family) |
| Parameters | 1.26B (hidden 2048, 28 layers, 32 Q / 8 KV heads, SwiGLU 4096) |
| Position encoding | RoPE (θ = 500,000) |
| Tokenizer | Byte-level BPE, 128,000 vocab |
| Context length | 4,096 |
| Precision | bf16 / fp16 |
| License | Apache-2.0 |

## Training

A three-stage, fully-documented pipeline on top of the from-scratch base:

1. **Continued pre-training** on a high-quality Korean mixture (FineWeb-2 `kor_Hang`,
   KOREAN-WEBTEXT, Korean Wikipedia), document-boundary packed.
2. **Encyclopedic annealing** on Korean Wikipedia (LR → 0) — the most token-efficient route to
   Korean knowledge.
3. **Supervised fine-tuning** on a 132K permissively-licensed Korean instruction mixture
   (KoAlpaca, OpenOrca-KO, KOpen-Platypus, KULLM-v2), with completion-only loss and explicit EOS
   supervision.

All continued-pretraining and instruction data are public corpora used only for post-training;
no external pretrained weights are used. A benchmark decontamination check found **0.00%** of benchmark
items substantially covered (≥50% of 25-character shingles) by the instruction data.

## Intended Use & Limitations

Intended for Korean text generation, QA, summarization, and research on small-model training.
As a compact model trained from scratch under a constrained budget, its **factual accuracy is
limited** and it can produce incorrect content; greedy decoding is best paired with a repetition
penalty. It trails much larger / higher-budget Korean models (e.g., EXAONE) on knowledge tasks
and has not undergone safety alignment. Use for research and non-critical applications only.

## Citation

```bibtex
@techreport{jumini2026,
  title  = {Jumini-Ko-1.2B Technical Report},
  author = {Cho, Ju-min},
  year   = {2026},
  note   = {https://huggingface.co/properly59/Jumini-Ko-1.2B}
}
```