76 lines
3.2 KiB
Markdown
76 lines
3.2 KiB
Markdown
---
|
||
language:
|
||
- ko
|
||
- en
|
||
license: apache-2.0
|
||
pipeline_tag: text-generation
|
||
tags:
|
||
- korean
|
||
- from-scratch
|
||
- qwen3-architecture
|
||
base_model:
|
||
- Qwen/Qwen3-1.7B
|
||
---
|
||
|
||
# Gaon-1.7B v2 Instruct (가온)
|
||
|
||
A bilingual (Korean + English) 1.7B chat model **trained entirely from scratch** —
|
||
architecture, data pipeline, pretraining, and instruction tuning — by one person on
|
||
borrowed idle GPUs, for $0. *Gaon* is pure Korean for "center/core."
|
||
|
||
- **Architecture:** Qwen3-1.7B-compatible (28L, hidden 2048, GQA 16Q/8KV, QK-Norm,
|
||
SwiGLU, tied embeddings, vocab 151,936). Loads with `Qwen3ForCausalLM`.
|
||
- **Pretraining:** 34B tokens (FineWeb-Edu EN + FineWeb-2 KO + ~20% Python code),
|
||
4× B200 FSDP, final loss 1.96.
|
||
- **Instruction tuning:** sequence-level distillation from Qwen2.5-7B-Instruct
|
||
(Apache-2.0), ~9k KO/EN instructions, SFT loss 1.26.
|
||
- **Code & full tech report:** https://github.com/k08200/gaon
|
||
|
||
**Run locally:** `ollama run hf.co/k08200/gaon-1.7b-v2-instruct-GGUF` — or community
|
||
GGUF quants (12 sizes, 516 MB – 3.4 GB, by [@mradermacher](https://huggingface.co/mradermacher)):
|
||
[static](https://huggingface.co/mradermacher/gaon-1.7b-v2-instruct-GGUF) ·
|
||
[imatrix](https://huggingface.co/mradermacher/gaon-1.7b-v2-instruct-i1-GGUF)
|
||
**Live demo:** [KO↔EN translator in your browser (WebGPU)](https://k08200.github.io/gaon/demo/)
|
||
|
||
## Usage
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
import torch
|
||
|
||
m = "k08200/gaon-1.7b-v2-instruct"
|
||
tok = AutoTokenizer.from_pretrained(m)
|
||
model = AutoModelForCausalLM.from_pretrained(m, torch_dtype=torch.bfloat16).eval()
|
||
|
||
msgs = [{"role": "user", "content": "한국의 수도는 어디인가요?"}]
|
||
prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
|
||
enc = tok(prompt, return_tensors="pt")
|
||
out = model.generate(**enc, max_new_tokens=200, do_sample=True, temperature=0.7)
|
||
print(tok.decode(out[0][enc.input_ids.shape[1]:], skip_special_tokens=True))
|
||
```
|
||
|
||
## Benchmarks
|
||
|
||
Base model (v2), lm-eval-harness, 0-shot, same settings for both:
|
||
|
||
| Benchmark | Gaon-1.7B v2 | Qwen3-1.7B-Base | Random |
|
||
|---|---|---|---|
|
||
| MMLU (English knowledge) | 25.1 | 62.6 | 25.0 |
|
||
| KMMLU (Korean knowledge) | 22.3 | 35.5 | 25.0 |
|
||
| HAERAE (Korean culture/lexis) | 19.9 | 46.8 | ~20 |
|
||
| KoBEST (Korean understanding) | 51.5 | — | ~50 |
|
||
|
||
This is the most honest number in the project: Gaon is **fluent in two languages,
|
||
follows instructions, and translates**, yet scores at *chance* on knowledge — because
|
||
Qwen3-1.7B (same architecture, same size) trained on ~36T tokens, roughly 1000× our
|
||
34B. **Linguistic competence emerges in tens of billions of tokens; world knowledge
|
||
needs trillions.** A continued-pretraining run (96B Korean-heavy tokens) lifts KMMLU
|
||
to ~27–29 then plateaus — see the [tech report](https://github.com/k08200/gaon/blob/main/docs/TECH_REPORT.md) §6.
|
||
|
||
## Honest limitations
|
||
|
||
A **from-scratch credential and research artifact**, not a frontier competitor.
|
||
Knowledge benchmarks at chance level (above); coding and strict-format tasks are weak.
|
||
Korean/English chat, explanation, and simple translation work well. Best used as a
|
||
reproducible small-LLM pipeline and a base for vertical fine-tuning on domain data.
|