初始化项目，由ModelHub XC社区提供模型

Model: properly59/Jumini-Ko-1.2B Source: Original Platform
2026-07-07 20:22:10 +08:00
commit 0d536ff270
16 changed files with 2105 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,37 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text
+Jumini-Ko-1.2B-Technical-Report.pdf filter=lfs diff=lfs merge=lfs -text
--- a/Jumini-Ko-1.2B-Technical-Report.pdf
+++ b/Jumini-Ko-1.2B-Technical-Report.pdf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0fd06737ba1b4f7844cd9b18cb9eec4b94c5d9c7871c80407d9d5687815f9fe8
+size 691101
--- a/17
+++ b/17
@@ -0,0 +1,17 @@
+Apache License
+Version 2.0, January 2004
+https://www.apache.org/licenses/
+
+Copyright 2026 Jumini-Ko HW2 contributors
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    https://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
--- a/README.md
+++ b/README.md
@@ -0,0 +1,141 @@
+---
+language:
+- ko
+license: apache-2.0
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- korean
+- causal-lm
+- decoder-only
+- from-scratch
+- instruction-tuned
+- 1.2b
+model-index:
+- name: Jumini-Ko-1.2B
+  results:
+  - task: {type: text-generation, name: Korean Knowledge (HAE-RAE Bench)}
+    dataset: {type: HAERAE-HUB/HAE_RAE_BENCH_1.0, name: HAE-RAE Bench}
+    metrics:
+    - {type: acc, name: accuracy (5-shot), value: 21.9}
+  - task: {type: text-generation, name: Korean Reading (Belebele-Ko)}
+    dataset: {type: facebook/belebele, name: Belebele (kor_Hang)}
+    metrics:
+    - {type: acc, name: accuracy (5-shot), value: 27.9}
+  - task: {type: text-generation, name: KMMLU}
+    dataset: {type: HAERAE-HUB/KMMLU, name: KMMLU}
+    metrics:
+    - {type: acc, name: accuracy (5-shot), value: 24.3}
+  - task: {type: text-generation, name: KoBEST}
+    dataset: {type: skt/kobest_v1, name: KoBEST}
+    metrics:
+    - {type: acc, name: accuracy (5-shot), value: 49.5}
+---
+
+# Jumini-Ko-1.2B
+
+**Jumini-Ko-1.2B** is a 1.26B-parameter Korean decoder-only language model **trained from
+scratch** — its architecture, tokenizer, data pipeline, and training loop were all built
+in-house, and it is *not* a fine-tune of any existing model. It is a compact,
+Korean-specialized model designed to run on commodity hardware.
+
+> Among the evaluated **open non-flagship Korean baselines** (`polyglot-ko-1.3b`, `Tri-1.9B`),
+> Jumini-Ko-1.2B is the **strongest on Korean knowledge (HAE-RAE) and reading comprehension
+> (Belebele-Ko)** — despite being the **smallest** model compared. The flagship
+> `EXAONE-4.0-1.2B`, trained on far more data/compute, is stronger on all four benchmarks.
+
+## Highlights
+
+- 🇰🇷 **Korean-specialized, from scratch** — Llama-3-style architecture (RoPE, GQA, SwiGLU,
+  RMSNorm), 128K byte-level BPE tokenizer, trained from random initialization.
+- 🥇 **Beats the size-matched `polyglot-ko-1.3b` and the larger `Tri-1.9B`** on HAE-RAE and
+  Belebele-Ko (5-shot), the two Korean-language benchmarks emphasized here. (It trails
+  `polyglot-ko-1.3b` on KoBEST commonsense and KMMLU, and the flagship `EXAONE-4.0-1.2B` overall.)
+- 🔬 **A data-centric recipe** — we show that *which* corpus you continue-pretrain on decides
+  *which* capability improves (web → commonsense, Wikipedia → knowledge).
+- 📦 **Edge-friendly** — 1.26B parameters; runs comfortably on a single consumer GPU.
+
+## Benchmark Results
+
+Korean benchmarks via the EleutherAI `lm-evaluation-harness`, 5-shot, accuracy (%). All models
+evaluated under identical settings. **Bold** = best, <u>underline</u> = second best.
+
+| Benchmark | **Jumini-Ko-1.2B** (1.26B) | polyglot-ko-1.3b (1.43B) | Tri-1.9B (1.9B) | EXAONE-4.0-1.2B† (1.28B) |
+|---|:--:|:--:|:--:|:--:|
+| HAE-RAE (Korean knowledge) | <u>21.9</u> | 18.7 | 18.9 | **30.0** |
+| Belebele-Ko (reading) | <u>27.9</u> | 22.4 | 22.9 | **44.7** |
+| KMMLU (knowledge) | 24.3 | <u>27.8</u> | 16.6 | **32.6** |
+| KoBEST (commonsense) | 49.5 | **55.9** | 50.1 | <u>50.6</u> |
+
+<sub>† EXAONE-4.0-1.2B is a strong flagship model trained on vastly more data/compute, shown as
+an aspirational reference. Against the **open same-tier** baselines (polyglot-ko-1.3b, Tri-1.9B),
+Jumini leads on the Korean-specific HAE-RAE and Belebele-Ko while being the smallest model.</sub>
+
+Jumini also beats `polyglot-ko-1.3b` on **4 of 5 HAE-RAE subtasks** (history, loan-word,
+rare-word, standard-nomenclature). It trails `polyglot-ko-1.3b` on commonsense (KoBEST) and broad
+knowledge (KMMLU). Full per-subtask numbers are in the technical report.
+
+## Quickstart
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+repo = "properly59/Jumini-Ko-1.2B"
+tok = AutoTokenizer.from_pretrained(repo)
+model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.float16, device_map="auto")
+
+prompt = "### 질문:\n대한민국의 수도는 어디인가요?\n\n### 답변:\n"
+ids = tok(tok.bos_token + prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
+out = model.generate(**ids, max_new_tokens=128, do_sample=True, temperature=0.8,
+                     min_p=0.05, repetition_penalty=1.2, no_repeat_ngram_size=3,
+                     pad_token_id=tok.pad_token_id)
+print(tok.decode(out[0][ids.input_ids.shape[1]:], skip_special_tokens=True))
+```
+
+## Model Details
+
+| | |
+|---|---|
+| Architecture | Decoder-only Transformer (Llama-3 family) |
+| Parameters | 1.26B (hidden 2048, 28 layers, 32 Q / 8 KV heads, SwiGLU 4096) |
+| Position encoding | RoPE (θ = 500,000) |
+| Tokenizer | Byte-level BPE, 128,000 vocab |
+| Context length | 4,096 |
+| Precision | bf16 / fp16 |
+| License | Apache-2.0 |
+
+## Training
+
+A three-stage, fully-documented pipeline on top of the from-scratch base:
+
+1. **Continued pre-training** on a high-quality Korean mixture (FineWeb-2 `kor_Hang`,
+   KOREAN-WEBTEXT, Korean Wikipedia), document-boundary packed.
+2. **Encyclopedic annealing** on Korean Wikipedia (LR → 0) — the most token-efficient route to
+   Korean knowledge.
+3. **Supervised fine-tuning** on a 132K permissively-licensed Korean instruction mixture
+   (KoAlpaca, OpenOrca-KO, KOpen-Platypus, KULLM-v2), with completion-only loss and explicit EOS
+   supervision.
+
+All continued-pretraining and instruction data are public corpora used only for post-training;
+no external pretrained weights are used. A benchmark decontamination check found **0.00%** of benchmark
+items substantially covered (≥50% of 25-character shingles) by the instruction data.
+
+## Intended Use & Limitations
+
+Intended for Korean text generation, QA, summarization, and research on small-model training.
+As a compact model trained from scratch under a constrained budget, its **factual accuracy is
+limited** and it can produce incorrect content; greedy decoding is best paired with a repetition
+penalty. It trails much larger / higher-budget Korean models (e.g., EXAONE) on knowledge tasks
+and has not undergone safety alignment. Use for research and non-critical applications only.
+
+## Citation
+
+```bibtex
+@techreport{jumini2026,
+  title  = {Jumini-Ko-1.2B Technical Report},
+  author = {Cho, Ju-min},
+  year   = {2026},
+  note   = {https://huggingface.co/properly59/Jumini-Ko-1.2B}
+}
+```
--- a/benchmark_results_20260609.json
+++ b/benchmark_results_20260609.json
@@ -0,0 +1,67 @@
+{
+  "benchmark_date": "2026-06-09",
+  "summary": "Local train-only Korean semantic sentinel diagnostics. These are not official leaderboard scores.",
+  "jumini_internal_scores_file": "jumini_internal_scores_20260609.json",
+  "main_public_comparison": {
+    "metric": "Korean semantic pass@33",
+    "prompt_set": "/tmp/chojm_hw2_runs/data/clean7020_passk32_accepted_imr_20260609/sentinel_imr_30x3_90.jsonl",
+    "decoding": {
+      "include_greedy_rollout": true,
+      "samples_per_prompt": 32,
+      "temperature": 0.7,
+      "top_p": 0.95,
+      "top_k": 50,
+      "max_new_tokens": 64
+    },
+    "models": {
+      "Jumini-Ko-1.2B-clean7020": {
+        "params": 1260505088,
+        "language_focus": "Korean from-scratch",
+        "summary_json": "/tmp/chojm_hw2_runs/passk_basin_inventory_20260609/clean7020_imr_k32_gpu2/passk_summary.json",
+        "pass_any": "38/90",
+        "pass_any_score": 42.22222222222222,
+        "pass_rollouts": "131/2884",
+        "pass_rollout_score": 4.542302357836339,
+        "greedy_hit": "0/90",
+        "categories": {
+          "instruction": "26/30",
+          "math": "8/30",
+          "reasoning": "4/30"
+        }
+      },
+      "TinyLlama-1.1B-intermediate-step-1431k-3T": {
+        "repo_id": "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T",
+        "params": 1100000000,
+        "language_focus": "English general base",
+        "summary_json": "/tmp/chojm_hw2_runs/hf_baseline_compare_20260609/tinyllama_1p1b_base_k32/summary.json",
+        "pass_any": "31/90",
+        "pass_any_score": 34.44444444444444,
+        "pass_rollouts": "56/2966",
+        "pass_rollout_score": 1.8880647336480108,
+        "greedy_hit": "0/90",
+        "categories": {
+          "instruction": "27/30",
+          "math": "4/30",
+          "reasoning": "0/30"
+        }
+      }
+    }
+  },
+  "public_refresh_evidence": {
+    "metric": "Stratified Korean semantic pass@k",
+    "previous_hf_v3a6200": {
+      "summary_json": "/tmp/chojm_hw2_runs/passk_basin_inventory_20260609/v3a6200_strat240/passk_summary.json",
+      "pass_any": "0/240",
+      "pass_any_score": 0.0,
+      "pass_rollouts": "0/2094",
+      "pass_rollout_score": 0.0
+    },
+    "current_clean7020": {
+      "summary_json": "/tmp/chojm_hw2_runs/passk_basin_inventory_20260609/clean7020_strat240_gpu2/passk_summary.json",
+      "pass_any": "21/240",
+      "pass_any_score": 8.75,
+      "pass_rollouts": "29/2105",
+      "pass_rollout_score": 1.3776722090261282
+    }
+  }
+}
--- a/config.json
+++ b/config.json
@@ -0,0 +1,32 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "dtype": "bfloat16",
+  "eos_token_id": 2,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 2048,
+  "initializer_range": 0.02,
+  "intermediate_size": 4096,
+  "max_position_embeddings": 4096,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 28,
+  "num_key_value_heads": 8,
+  "pad_token_id": 0,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_parameters": {
+    "rope_theta": 500000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.12.0",
+  "use_cache": false,
+  "vocab_size": 128000
+}
--- a/generation_config.json
+++ b/generation_config.json
@@ -0,0 +1,9 @@
+{
+  "bos_token_id": 1,
+  "do_sample": true,
+  "eos_token_id": 2,
+  "pad_token_id": 0,
+  "temperature": 0.8,
+  "top_k": 50,
+  "transformers_version": "5.12.0"
+}
--- a/hf_equivalence_atol2e-5.json
+++ b/hf_equivalence_atol2e-5.json
@@ -0,0 +1,30 @@
+{
+  "ok": true,
+  "logits_shape": [
+    2,
+    8,
+    128000
+  ],
+  "max_abs_diff": 1.4781951904296875e-05,
+  "mean_abs_diff": 1.073120984074194e-06,
+  "rms_diff": 1.5380918512164499e-06,
+  "max_rel_diff": 0.04641776159405708,
+  "argmax_mismatch_rate": 0.0,
+  "atol": 2e-05,
+  "rtol": 2e-05,
+  "config": "/raid2/chojm/hw2_llm_from_scratch/configs/jumini_1_2b.json",
+  "checkpoint": "/tmp/chojm_hw2_runs/jumini-ko-1.2b-firsttok7000-argmax-boundary-layernormshock-v1-last16-4gpu0123-mb8-probe25-rerun20260608/step_00007020/model.pt",
+  "hf_model_dir": "/raid2/chojm/hw2_llm_from_scratch/models/jumini-ko-1.2b-clean7020-best-20260609-hf-fp32",
+  "hf_model_class": "LlamaForCausalLM",
+  "device": "cpu",
+  "comparison_dtype": "float32",
+  "hf_export_torch_dtype": "float32",
+  "round_native_to_export_dtype": false,
+  "input_shape": [
+    2,
+    8
+  ],
+  "input_ids_sha256": "b9225a84aaeeb49dbb7ac34866e25a8fdbf0823d9fe02530e3686d2e09f72f6f",
+  "seed": 1234,
+  "max_tokens": 64
+}
--- a/jumini_config.json
+++ b/jumini_config.json
@@ -0,0 +1,23 @@
+{
+  "vocab_size": 128000,
+  "hidden_size": 2048,
+  "intermediate_size": 4096,
+  "num_hidden_layers": 28,
+  "num_attention_heads": 32,
+  "num_key_value_heads": 8,
+  "max_position_embeddings": 4096,
+  "rope_theta": 500000.0,
+  "rms_norm_eps": 1e-05,
+  "initializer_range": 0.02,
+  "tie_word_embeddings": true,
+  "attention_dropout": 0.0,
+  "residual_dropout": 0.0,
+  "gradient_checkpointing": true,
+  "pad_token_id": 0,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "model_type": "jumini",
+  "architectures": [
+    "JuminiForCausalLM"
+  ]
+}
--- a/jumini_internal_scores_20260609.json
+++ b/jumini_internal_scores_20260609.json
--- a/load_smoke.json
+++ b/load_smoke.json
@@ -0,0 +1,19 @@
+{
+  "model_dir": "models/jumini-ko-1.2b-clean7020-best-20260609-hf-fp32",
+  "load_model": true,
+  "model_type": "llama",
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "vocab_size": 128000,
+  "hidden_size": 2048,
+  "num_hidden_layers": 28,
+  "tokenizer_len": 128000,
+  "bos_token": "[BOS]",
+  "eos_token": "[EOS]",
+  "pad_token": "[PAD]",
+  "model_class": "LlamaForCausalLM",
+  "params": 1260505088,
+  "parameter_count": 1260505088,
+  "ok": true
+}
--- a/model.safetensors
+++ b/model.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6e355c9f2c5e6f8f5c8b9074c5ea944838eb9728b86c7abd7ac26c460fcd3bf1
+size 2521039488
--- a/replacement_evidence_20260609.json
+++ b/replacement_evidence_20260609.json
@@ -0,0 +1,32 @@
+{
+  "repo_id": "properly59/Jumini-Ko-1.2B",
+  "replacement_date": "2026-06-09",
+  "previous_hf_checkpoint": {
+    "name": "v3a6200",
+    "remote_sha_before_replace": "369600b6f14950e4bf6c7c43457ed3ee0921663c",
+    "checkpoint": "/raid2/chojm/hw2_llm_from_scratch/experiments/jumini-ko-1.2b-phase2-mixture-v3a-retention-continue-5700-to-6200-2gpu12-mb9/step_00006200/model.pt",
+    "strat240_pass_any": "0/240",
+    "strat240_pass_rollouts": "0/2094",
+    "strat240_mechanical_bad_rollouts": "1755/2094"
+  },
+  "new_checkpoint": {
+    "name": "clean7020",
+    "checkpoint": "/tmp/chojm_hw2_runs/jumini-ko-1.2b-firsttok7000-argmax-boundary-layernormshock-v1-last16-4gpu0123-mb8-probe25-rerun20260608/step_00007020/model.pt",
+    "export_dir": "models/jumini-ko-1.2b-clean7020-best-20260609-hf-fp32",
+    "strat240_pass_any": "21/240",
+    "strat240_pass_rollouts": "29/2105",
+    "strat240_mechanical_bad_rollouts": "856/2105",
+    "k32_pass_any": "38/90",
+    "strict_heldout_stable_semantic_hits": "2/12"
+  },
+  "validation": {
+    "load_smoke": "models/jumini-ko-1.2b-clean7020-best-20260609-hf-fp32/load_smoke.json",
+    "hf_equivalence": "models/jumini-ko-1.2b-clean7020-best-20260609-hf-fp32/hf_equivalence_atol2e-5.json",
+    "hf_equivalence_argmax_mismatch_rate": 0.0,
+    "hf_equivalence_max_abs_diff": 0.000014781951904296875
+  },
+  "notes": [
+    "This is a pragmatic replacement over the existing HF upload because local average diagnostics are better than v3a6200.",
+    "Strict heldout remains unsolved; this is not labeled as a final strong-candidate breakthrough."
+  ]
+}
--- a/special_tokens_map.json
+++ b/special_tokens_map.json
@@ -0,0 +1,6 @@
+{
+  "bos_token": "[BOS]",
+  "eos_token": "[EOS]",
+  "unk_token": "[UNK]",
+  "pad_token": "[PAD]"
+}
--- a/tokenizer.json
+++ b/tokenizer.json
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:02a1d2e971468994add44673bd84e6ca77735752515a402e196d12336d1283ac
+size 11624392
--- a/tokenizer_config.json
+++ b/tokenizer_config.json
@@ -0,0 +1,12 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "[BOS]",
+  "eos_token": "[EOS]",
+  "fix_mistral_regex": true,
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 4096,
+  "pad_token": "[PAD]",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "[UNK]"
+}