--- language: - ko license: apache-2.0 library_name: transformers pipeline_tag: text-generation tags: - korean - causal-lm - decoder-only - from-scratch - instruction-tuned - 1.2b model-index: - name: Jumini-Ko-1.2B results: - task: {type: text-generation, name: Korean Knowledge (HAE-RAE Bench)} dataset: {type: HAERAE-HUB/HAE_RAE_BENCH_1.0, name: HAE-RAE Bench} metrics: - {type: acc, name: accuracy (5-shot), value: 21.9} - task: {type: text-generation, name: Korean Reading (Belebele-Ko)} dataset: {type: facebook/belebele, name: Belebele (kor_Hang)} metrics: - {type: acc, name: accuracy (5-shot), value: 27.9} - task: {type: text-generation, name: KMMLU} dataset: {type: HAERAE-HUB/KMMLU, name: KMMLU} metrics: - {type: acc, name: accuracy (5-shot), value: 24.3} - task: {type: text-generation, name: KoBEST} dataset: {type: skt/kobest_v1, name: KoBEST} metrics: - {type: acc, name: accuracy (5-shot), value: 49.5} --- # Jumini-Ko-1.2B **Jumini-Ko-1.2B** is a 1.26B-parameter Korean decoder-only language model **trained from scratch** β€” its architecture, tokenizer, data pipeline, and training loop were all built in-house, and it is *not* a fine-tune of any existing model. It is a compact, Korean-specialized model designed to run on commodity hardware. > Among the evaluated **open non-flagship Korean baselines** (`polyglot-ko-1.3b`, `Tri-1.9B`), > Jumini-Ko-1.2B is the **strongest on Korean knowledge (HAE-RAE) and reading comprehension > (Belebele-Ko)** β€” despite being the **smallest** model compared. The flagship > `EXAONE-4.0-1.2B`, trained on far more data/compute, is stronger on all four benchmarks. ## Highlights - πŸ‡°πŸ‡· **Korean-specialized, from scratch** β€” Llama-3-style architecture (RoPE, GQA, SwiGLU, RMSNorm), 128K byte-level BPE tokenizer, trained from random initialization. - πŸ₯‡ **Beats the size-matched `polyglot-ko-1.3b` and the larger `Tri-1.9B`** on HAE-RAE and Belebele-Ko (5-shot), the two Korean-language benchmarks emphasized here. (It trails `polyglot-ko-1.3b` on KoBEST commonsense and KMMLU, and the flagship `EXAONE-4.0-1.2B` overall.) - πŸ”¬ **A data-centric recipe** β€” we show that *which* corpus you continue-pretrain on decides *which* capability improves (web β†’ commonsense, Wikipedia β†’ knowledge). - πŸ“¦ **Edge-friendly** β€” 1.26B parameters; runs comfortably on a single consumer GPU. ## Benchmark Results Korean benchmarks via the EleutherAI `lm-evaluation-harness`, 5-shot, accuracy (%). All models evaluated under identical settings. **Bold** = best, underline = second best. | Benchmark | **Jumini-Ko-1.2B** (1.26B) | polyglot-ko-1.3b (1.43B) | Tri-1.9B (1.9B) | EXAONE-4.0-1.2B† (1.28B) | |---|:--:|:--:|:--:|:--:| | HAE-RAE (Korean knowledge) | 21.9 | 18.7 | 18.9 | **30.0** | | Belebele-Ko (reading) | 27.9 | 22.4 | 22.9 | **44.7** | | KMMLU (knowledge) | 24.3 | 27.8 | 16.6 | **32.6** | | KoBEST (commonsense) | 49.5 | **55.9** | 50.1 | 50.6 | † EXAONE-4.0-1.2B is a strong flagship model trained on vastly more data/compute, shown as an aspirational reference. Against the **open same-tier** baselines (polyglot-ko-1.3b, Tri-1.9B), Jumini leads on the Korean-specific HAE-RAE and Belebele-Ko while being the smallest model. Jumini also beats `polyglot-ko-1.3b` on **4 of 5 HAE-RAE subtasks** (history, loan-word, rare-word, standard-nomenclature). It trails `polyglot-ko-1.3b` on commonsense (KoBEST) and broad knowledge (KMMLU). Full per-subtask numbers are in the technical report. ## Quickstart ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer repo = "properly59/Jumini-Ko-1.2B" tok = AutoTokenizer.from_pretrained(repo) model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.float16, device_map="auto") prompt = "### 질문:\nλŒ€ν•œλ―Όκ΅­μ˜ μˆ˜λ„λŠ” μ–΄λ””μΈκ°€μš”?\n\n### λ‹΅λ³€:\n" ids = tok(tok.bos_token + prompt, return_tensors="pt", add_special_tokens=False).to(model.device) out = model.generate(**ids, max_new_tokens=128, do_sample=True, temperature=0.8, min_p=0.05, repetition_penalty=1.2, no_repeat_ngram_size=3, pad_token_id=tok.pad_token_id) print(tok.decode(out[0][ids.input_ids.shape[1]:], skip_special_tokens=True)) ``` ## Model Details | | | |---|---| | Architecture | Decoder-only Transformer (Llama-3 family) | | Parameters | 1.26B (hidden 2048, 28 layers, 32 Q / 8 KV heads, SwiGLU 4096) | | Position encoding | RoPE (ΞΈ = 500,000) | | Tokenizer | Byte-level BPE, 128,000 vocab | | Context length | 4,096 | | Precision | bf16 / fp16 | | License | Apache-2.0 | ## Training A three-stage, fully-documented pipeline on top of the from-scratch base: 1. **Continued pre-training** on a high-quality Korean mixture (FineWeb-2 `kor_Hang`, KOREAN-WEBTEXT, Korean Wikipedia), document-boundary packed. 2. **Encyclopedic annealing** on Korean Wikipedia (LR β†’ 0) β€” the most token-efficient route to Korean knowledge. 3. **Supervised fine-tuning** on a 132K permissively-licensed Korean instruction mixture (KoAlpaca, OpenOrca-KO, KOpen-Platypus, KULLM-v2), with completion-only loss and explicit EOS supervision. All continued-pretraining and instruction data are public corpora used only for post-training; no external pretrained weights are used. A benchmark decontamination check found **0.00%** of benchmark items substantially covered (β‰₯50% of 25-character shingles) by the instruction data. ## Intended Use & Limitations Intended for Korean text generation, QA, summarization, and research on small-model training. As a compact model trained from scratch under a constrained budget, its **factual accuracy is limited** and it can produce incorrect content; greedy decoding is best paired with a repetition penalty. It trails much larger / higher-budget Korean models (e.g., EXAONE) on knowledge tasks and has not undergone safety alignment. Use for research and non-critical applications only. ## Citation ```bibtex @techreport{jumini2026, title = {Jumini-Ko-1.2B Technical Report}, author = {Cho, Ju-min}, year = {2026}, note = {https://huggingface.co/properly59/Jumini-Ko-1.2B} } ```