初始化项目，由ModelHub XC社区提供模型
Model: properly59/Jumini-Ko-1.2B Source: Original Platform
This commit is contained in:
ModelHub XC
141
README.md
Normal file
141
README.md
Normal file
@@ -0,0 +1,141 @@
|
||||
---
|
||||
language:
|
||||
- ko
|
||||
license: apache-2.0
|
||||
library_name: transformers
|
||||
pipeline_tag: text-generation
|
||||
tags:
|
||||
- korean
|
||||
- causal-lm
|
||||
- decoder-only
|
||||
- from-scratch
|
||||
- instruction-tuned
|
||||
- 1.2b
|
||||
model-index:
|
||||
- name: Jumini-Ko-1.2B
|
||||
results:
|
||||
- task: {type: text-generation, name: Korean Knowledge (HAE-RAE Bench)}
|
||||
dataset: {type: HAERAE-HUB/HAE_RAE_BENCH_1.0, name: HAE-RAE Bench}
|
||||
metrics:
|
||||
- {type: acc, name: accuracy (5-shot), value: 21.9}
|
||||
- task: {type: text-generation, name: Korean Reading (Belebele-Ko)}
|
||||
dataset: {type: facebook/belebele, name: Belebele (kor_Hang)}
|
||||
metrics:
|
||||
- {type: acc, name: accuracy (5-shot), value: 27.9}
|
||||
- task: {type: text-generation, name: KMMLU}
|
||||
dataset: {type: HAERAE-HUB/KMMLU, name: KMMLU}
|
||||
metrics:
|
||||
- {type: acc, name: accuracy (5-shot), value: 24.3}
|
||||
- task: {type: text-generation, name: KoBEST}
|
||||
dataset: {type: skt/kobest_v1, name: KoBEST}
|
||||
metrics:
|
||||
- {type: acc, name: accuracy (5-shot), value: 49.5}
|
||||
---
|
||||
|
||||
# Jumini-Ko-1.2B
|
||||
|
||||
**Jumini-Ko-1.2B** is a 1.26B-parameter Korean decoder-only language model **trained from
|
||||
scratch** — its architecture, tokenizer, data pipeline, and training loop were all built
|
||||
in-house, and it is *not* a fine-tune of any existing model. It is a compact,
|
||||
Korean-specialized model designed to run on commodity hardware.
|
||||
|
||||
> Among the evaluated **open non-flagship Korean baselines** (`polyglot-ko-1.3b`, `Tri-1.9B`),
|
||||
> Jumini-Ko-1.2B is the **strongest on Korean knowledge (HAE-RAE) and reading comprehension
|
||||
> (Belebele-Ko)** — despite being the **smallest** model compared. The flagship
|
||||
> `EXAONE-4.0-1.2B`, trained on far more data/compute, is stronger on all four benchmarks.
|
||||
|
||||
## Highlights
|
||||
|
||||
- 🇰🇷 **Korean-specialized, from scratch** — Llama-3-style architecture (RoPE, GQA, SwiGLU,
|
||||
RMSNorm), 128K byte-level BPE tokenizer, trained from random initialization.
|
||||
- 🥇 **Beats the size-matched `polyglot-ko-1.3b` and the larger `Tri-1.9B`** on HAE-RAE and
|
||||
Belebele-Ko (5-shot), the two Korean-language benchmarks emphasized here. (It trails
|
||||
`polyglot-ko-1.3b` on KoBEST commonsense and KMMLU, and the flagship `EXAONE-4.0-1.2B` overall.)
|
||||
- 🔬 **A data-centric recipe** — we show that *which* corpus you continue-pretrain on decides
|
||||
*which* capability improves (web → commonsense, Wikipedia → knowledge).
|
||||
- 📦 **Edge-friendly** — 1.26B parameters; runs comfortably on a single consumer GPU.
|
||||
|
||||
## Benchmark Results
|
||||
|
||||
Korean benchmarks via the EleutherAI `lm-evaluation-harness`, 5-shot, accuracy (%). All models
|
||||
evaluated under identical settings. **Bold** = best, <u>underline</u> = second best.
|
||||
|
||||
| Benchmark | **Jumini-Ko-1.2B** (1.26B) | polyglot-ko-1.3b (1.43B) | Tri-1.9B (1.9B) | EXAONE-4.0-1.2B† (1.28B) |
|
||||
|---|:--:|:--:|:--:|:--:|
|
||||
| HAE-RAE (Korean knowledge) | <u>21.9</u> | 18.7 | 18.9 | **30.0** |
|
||||
| Belebele-Ko (reading) | <u>27.9</u> | 22.4 | 22.9 | **44.7** |
|
||||
| KMMLU (knowledge) | 24.3 | <u>27.8</u> | 16.6 | **32.6** |
|
||||
| KoBEST (commonsense) | 49.5 | **55.9** | 50.1 | <u>50.6</u> |
|
||||
|
||||
<sub>† EXAONE-4.0-1.2B is a strong flagship model trained on vastly more data/compute, shown as
|
||||
an aspirational reference. Against the **open same-tier** baselines (polyglot-ko-1.3b, Tri-1.9B),
|
||||
Jumini leads on the Korean-specific HAE-RAE and Belebele-Ko while being the smallest model.</sub>
|
||||
|
||||
Jumini also beats `polyglot-ko-1.3b` on **4 of 5 HAE-RAE subtasks** (history, loan-word,
|
||||
rare-word, standard-nomenclature). It trails `polyglot-ko-1.3b` on commonsense (KoBEST) and broad
|
||||
knowledge (KMMLU). Full per-subtask numbers are in the technical report.
|
||||
|
||||
## Quickstart
|
||||
|
||||
```python
|
||||
import torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
repo = "properly59/Jumini-Ko-1.2B"
|
||||
tok = AutoTokenizer.from_pretrained(repo)
|
||||
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.float16, device_map="auto")
|
||||
|
||||
prompt = "### 질문:\n대한민국의 수도는 어디인가요?\n\n### 답변:\n"
|
||||
ids = tok(tok.bos_token + prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
|
||||
out = model.generate(**ids, max_new_tokens=128, do_sample=True, temperature=0.8,
|
||||
min_p=0.05, repetition_penalty=1.2, no_repeat_ngram_size=3,
|
||||
pad_token_id=tok.pad_token_id)
|
||||
print(tok.decode(out[0][ids.input_ids.shape[1]:], skip_special_tokens=True))
|
||||
```
|
||||
|
||||
## Model Details
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Architecture | Decoder-only Transformer (Llama-3 family) |
|
||||
| Parameters | 1.26B (hidden 2048, 28 layers, 32 Q / 8 KV heads, SwiGLU 4096) |
|
||||
| Position encoding | RoPE (θ = 500,000) |
|
||||
| Tokenizer | Byte-level BPE, 128,000 vocab |
|
||||
| Context length | 4,096 |
|
||||
| Precision | bf16 / fp16 |
|
||||
| License | Apache-2.0 |
|
||||
|
||||
## Training
|
||||
|
||||
A three-stage, fully-documented pipeline on top of the from-scratch base:
|
||||
|
||||
1. **Continued pre-training** on a high-quality Korean mixture (FineWeb-2 `kor_Hang`,
|
||||
KOREAN-WEBTEXT, Korean Wikipedia), document-boundary packed.
|
||||
2. **Encyclopedic annealing** on Korean Wikipedia (LR → 0) — the most token-efficient route to
|
||||
Korean knowledge.
|
||||
3. **Supervised fine-tuning** on a 132K permissively-licensed Korean instruction mixture
|
||||
(KoAlpaca, OpenOrca-KO, KOpen-Platypus, KULLM-v2), with completion-only loss and explicit EOS
|
||||
supervision.
|
||||
|
||||
All continued-pretraining and instruction data are public corpora used only for post-training;
|
||||
no external pretrained weights are used. A benchmark decontamination check found **0.00%** of benchmark
|
||||
items substantially covered (≥50% of 25-character shingles) by the instruction data.
|
||||
|
||||
## Intended Use & Limitations
|
||||
|
||||
Intended for Korean text generation, QA, summarization, and research on small-model training.
|
||||
As a compact model trained from scratch under a constrained budget, its **factual accuracy is
|
||||
limited** and it can produce incorrect content; greedy decoding is best paired with a repetition
|
||||
penalty. It trails much larger / higher-budget Korean models (e.g., EXAONE) on knowledge tasks
|
||||
and has not undergone safety alignment. Use for research and non-critical applications only.
|
||||
|
||||
## Citation
|
||||
|
||||
```bibtex
|
||||
@techreport{jumini2026,
|
||||
title = {Jumini-Ko-1.2B Technical Report},
|
||||
author = {Cho, Ju-min},
|
||||
year = {2026},
|
||||
note = {https://huggingface.co/properly59/Jumini-Ko-1.2B}
|
||||
}
|
||||
```
|
||||
Reference in New Issue
Block a user