初始化项目，由ModelHub XC社区提供模型

Model: NotoriousH2/gemma-3-1b-it-Math-GRPO Source: Original Platform
2026-07-03 06:44:16 +08:00
commit 908e4255d5
17 changed files with 52204 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text
--- a/README.md
+++ b/README.md
@@ -0,0 +1,81 @@
+---
+language: ko
+license: apache-2.0
+base_model: google/gemma-3-1b-it
+tags:
+- math
+- korean
+- grpo
+- rl
+- gemma
+datasets:
+- NotoriousH2/HRM8K
+---
+
+# Gemma-3-1B-IT Math GRPO
+
+SFT → RS-SFT → GRPO 3단계 파이프라인으로 학습한 한국어 수학 모델.
+
+## 성능
+
+| Benchmark | Score |
+|-----------|-------|
+| HRM8K eval GSM8K (264문제, Korean) | **~46.2%** |
+| HRM8K eval MATH (577문제, Korean) | ~16.5% |
+
+> ⚠️ GRPO는 base (RS-SFT, ~46.6%)대비 유의미한 개선 없음. 이미 SFT+RS-SFT로 최적화된 1B 모델에서 RL 추가 여지가 극히 제한적.
+
+## 데이터 & 학습 파이프라인
+
+### Stage 1-2: SFT → RS-SFT
+위 RS-SFT 모델과 동일. (SFT 교사 증류 → RS 샘플링 → RS-SFT with 5x replay)
+
+### Stage 3: GRPO
+
+#### Reward 함수
+
+
+#### 학습 데이터
+- **프롬프트만 필요** (GRPO는 학습 중 자체 생성)
+- GSM8K train 6,871개 unique 한국어 문제
+- 각 문제에 ground truth 답 포함 (reward 계산용)
+
+#### 학습 설정
+
+
+## DPO vs GRPO 비교 (실험 결과)
+
+### DPO 실패 분석 (10회)
+| 데이터 전략 | GSM8K | 분석 |
+|------------|-------|------|
+| 기존 방식 (shortest correct + longest incorrect) | 48.1% | 길이 편향만 학습 |
+| Length-matched (55자 차이) | 46.2% | 신호 없음 (DPO acc≈0.50) |
+| Teacher-chosen (30B 교사 풀이=chosen) | 47.3% | OOD 문제 |
+| Multi-pair (질문당 3쌍, 난이도 가중) | 46.6% | 양 증가도 무효 |
+| base 대비 | ±0-2%p | 모두 variance 범위 |
+
+**DPO 근본 문제**: 1B 모델이 정답/오답 풀이의 미묘한 차이를 내부적으로 구분할 capacity 부족.
+
+### GRPO 결과 (2회)
+| beta | steps | GSM8K | 비고 |
+|------|-------|-------|------|
+| 0.001 | 200 | 43.9% | format 퇴보 (boxed↓) |
+| 0.04 | 500 | 46.2% | base와 동일 수준 |
+
+## 환경
+- GPU: H100 NVL 95GB
+- Framework: trl 0.29.0, transformers 4.57.3, vllm 0.11.0
+- GRPO 학습: ~55분 (vLLM colocate 사용)
+
+## 재현 방법
+
+INFO 03-19 14:53:37 [__init__.py:216] Automatically detected platform cuda.
+[1;36m(APIServer pid=3429210)[0;0m INFO 03-19 14:53:43 [api_server.py:1839] vLLM API server version 0.11.0
+[1;36m(APIServer pid=3429210)[0;0m INFO 03-19 14:53:43 [utils.py:233] non-default args: {'model_tag': './grpo_model', 'model': './grpo_model', 'dtype': 'bfloat16', 'max_model_len': 4096, 'gpu_memory_utilization': 0.85}
+
+## 파일
+- : Stage 1 SFT
+- : RS 샘플링
+- : Stage 2 RS-SFT
+- : Stage 3 GRPO
+- : HRM8K 평가
--- a/added_tokens.json
+++ b/added_tokens.json
@@ -0,0 +1,3 @@
+{
+  "<image_soft_token>": 262144
+}
--- a/chat_template.jinja
+++ b/chat_template.jinja
@@ -0,0 +1,47 @@
+{{ bos_token }}
+{%- if messages[0]['role'] == 'system' -%}
+    {%- if messages[0]['content'] is string -%}
+        {%- set first_user_prefix = messages[0]['content'] + '
+
+' -%}
+    {%- else -%}
+        {%- set first_user_prefix = messages[0]['content'][0]['text'] + '
+
+' -%}
+    {%- endif -%}
+    {%- set loop_messages = messages[1:] -%}
+{%- else -%}
+    {%- set first_user_prefix = "" -%}
+    {%- set loop_messages = messages -%}
+{%- endif -%}
+{%- for message in loop_messages -%}
+    {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}
+        {{ raise_exception("Conversation roles must alternate user/assistant/user/assistant/...") }}
+    {%- endif -%}
+    {%- if (message['role'] == 'assistant') -%}
+        {%- set role = "model" -%}
+    {%- else -%}
+        {%- set role = message['role'] -%}
+    {%- endif -%}
+    {{ '<start_of_turn>' + role + '
+' + (first_user_prefix if loop.first else "") }}
+    {%- if message['content'] is string -%}
+        {{ message['content'] | trim }}
+    {%- elif message['content'] is iterable -%}
+        {%- for item in message['content'] -%}
+            {%- if item['type'] == 'image' -%}
+                {{ '<start_of_image>' }}
+            {%- elif item['type'] == 'text' -%}
+                {{ item['text'] | trim }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{ raise_exception("Invalid content type") }}
+    {%- endif -%}
+    {{ '<end_of_turn>
+' }}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    {{'<start_of_turn>model
+'}}
+{%- endif -%}
--- a/config.json
+++ b/config.json
@@ -0,0 +1,64 @@
+{
+  "_sliding_window_pattern": 6,
+  "architectures": [
+    "Gemma3ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "attn_logit_softcapping": null,
+  "bos_token_id": 2,
+  "cache_implementation": "hybrid",
+  "dtype": "bfloat16",
+  "eos_token_id": 1,
+  "final_logit_softcapping": null,
+  "head_dim": 256,
+  "hidden_activation": "gelu_pytorch_tanh",
+  "hidden_size": 1152,
+  "initializer_range": 0.02,
+  "intermediate_size": 6912,
+  "layer_types": [
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention"
+  ],
+  "max_position_embeddings": 32768,
+  "model_type": "gemma3_text",
+  "num_attention_heads": 4,
+  "num_hidden_layers": 26,
+  "num_key_value_heads": 1,
+  "pad_token_id": 1,
+  "query_pre_attn_scalar": 256,
+  "rms_norm_eps": 1e-06,
+  "rope_local_base_freq": 10000,
+  "rope_scaling": null,
+  "rope_theta": 1000000,
+  "sliding_window": 512,
+  "sliding_window_pattern": 6,
+  "transformers_version": "4.57.3",
+  "use_bidirectional_attention": false,
+  "use_cache": false,
+  "vocab_size": 262144
+}
--- a/eval.py
+++ b/eval.py
@@ -0,0 +1,86 @@
+"""공통 평가 스크립트: vLLM 서버에 연결하여 HRM8K 전체 841문제 평가 (temperature=0)"""
+import os, json, re, sys, asyncio
+from langchain_openai import ChatOpenAI
+from langchain_core.prompts import ChatPromptTemplate
+from langchain_core.output_parsers import StrOutputParser
+from openai import OpenAI
+
+MATH_SYSTEM_PROMPT = """주어진 수학 문제를 단계별로 풀고 답변을 작성하세요.
+반드시 최종 답변을 \\boxed{정수} 형식으로 마지막 줄에 출력하세요.
+예시: \\boxed{42}"""
+
+def extract_boxed(text):
+    m = re.findall(r'\\boxed\{([^}]+)\}', text)
+    return m[-1].strip() if m else None
+
+def normalize(a):
+    if a is None: return None
+    s = str(a).replace(",","").replace(" ","").strip()
+    try:
+        n = float(s)
+        return str(int(n)) if n == int(n) else str(n)
+    except: return s
+
+def check(pred, gt):
+    p, g = normalize(pred), normalize(gt)
+    return p is not None and g is not None and p == g
+
+async def evaluate(label="", save_path=None):
+    client = OpenAI(base_url="http://localhost:8000/v1", api_key="token-abc123")
+    model_name = client.models.list().data[0].id
+    print(f"모델: {model_name}")
+
+    with open("data/HRM8k_eval.json") as f:
+        data = json.load(f)
+    print(f"평가: {len(data)}개 (temperature=0, max_tokens=2048)")
+
+    llm = ChatOpenAI(base_url="http://localhost:8000/v1", api_key="token-abc123",
+                     model=model_name, temperature=0, max_tokens=2048)
+    prompt = ChatPromptTemplate([("user", "{sp}\n\n{q}")]).partial(sp=MATH_SYSTEM_PROMPT)
+    chain = prompt | llm | StrOutputParser()
+    inputs = [{"q": item["question"]} for item in data]
+    results = await chain.abatch(inputs, config={"max_concurrency": 400})
+
+    by_src = {}
+    details = []
+    for item, res in zip(data, results):
+        s = item.get("source", "?")
+        if s not in by_src: by_src[s] = {"correct": 0, "total": 0, "no_boxed": 0}
+        by_src[s]["total"] += 1
+        pred = extract_boxed(res)
+        is_correct = False
+        if pred is None:
+            by_src[s]["no_boxed"] += 1
+        elif check(pred, item["answer"]):
+            by_src[s]["correct"] += 1
+            is_correct = True
+        details.append({
+            "question": item["question"][:80],
+            "source": s,
+            "gt": str(item["answer"])[-30:] if isinstance(item["answer"], str) else str(item["answer"]),
+            "pred": pred,
+            "correct": is_correct,
+        })
+
+    tc = sum(v["correct"] for v in by_src.values())
+    tt = sum(v["total"] for v in by_src.values())
+    print(f"\n=== {label} 결과 (temperature=0) ===")
+    for s in sorted(by_src):
+        v = by_src[s]
+        print(f"  [{s.upper()}] {v['correct']}/{v['total']} ({v['correct']/v['total']*100:.1f}%) | boxed미출력: {v['no_boxed']}")
+    print(f"  [전체] {tc}/{tt} ({tc/tt*100:.1f}%)")
+
+    result_obj = {"label": label, "correct": tc, "total": tt, "accuracy": tc/tt*100, "by_source": by_src}
+
+    if save_path:
+        os.makedirs(os.path.dirname(save_path) or ".", exist_ok=True)
+        with open(save_path, "w") as f:
+            json.dump({"result": result_obj, "details": details}, f, ensure_ascii=False, indent=2)
+        print(f"  결과 저장: {save_path}")
+
+    return result_obj
+
+if __name__ == "__main__":
+    label = sys.argv[1] if len(sys.argv) > 1 else "eval"
+    save_path = sys.argv[2] if len(sys.argv) > 2 else None
+    asyncio.run(evaluate(label, save_path))
--- a/generation_config.json
+++ b/generation_config.json
@@ -0,0 +1,14 @@
+{
+  "bos_token_id": 2,
+  "cache_implementation": "hybrid",
+  "do_sample": true,
+  "eos_token_id": [
+    1,
+    1,
+    106
+  ],
+  "pad_token_id": 1,
+  "top_k": 64,
+  "top_p": 0.95,
+  "transformers_version": "4.57.3"
+}
--- a/model.safetensors
+++ b/model.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f8e51950c5d8c24f07430aa6c6a9a4900d01825d0e1fc8870561375fa6684295
+size 1999811208
--- a/rs_sample.py
+++ b/rs_sample.py
@@ -0,0 +1,108 @@
+"""RS sampling from C18-2 model (48.5% GSM8K)"""
+import json, re, asyncio, random
+from openai import AsyncOpenAI
+
+SP = "주어진 수학 문제를 단계별로 풀고 답변을 작성하세요.\n반드시 최종 답변을 \\boxed{정수} 형식으로 마지막 줄에 출력하세요.\n예시: \\boxed{42}"
+
+def extract_boxed(text):
+    m = re.findall(r'\\boxed\{([^}]+)\}', text)
+    return m[-1].strip() if m else None
+
+def normalize(a):
+    if a is None: return None
+    s = str(a).replace(",","").replace(" ","").strip()
+    try:
+        n = float(s)
+        return str(int(n)) if n == int(n) else str(n)
+    except: return s
+
+# Load GSM8K train questions (the ones we have gold answers for)
+with open("data/GSM8K_full_qwen3_30b.json") as f:
+    data = json.load(f)
+
+# Get unique questions with their gold answers
+q_to_gold = {}
+for d in data:
+    q = d["question"]
+    if q not in q_to_gold:
+        # Extract gold from the existing correct solutions
+        gold = extract_boxed(d["answer"])
+        if gold:
+            q_to_gold[q] = normalize(gold)
+
+questions = list(q_to_gold.keys())
+random.seed(42)
+random.shuffle(questions)
+# Sample a subset for RS (use all unique questions)
+print(f"Total unique questions: {len(questions)}")
+
+client = AsyncOpenAI(base_url="http://localhost:8000/v1", api_key="token-abc123")
+
+async def sample_one(question, n=16):
+    messages = [{"role": "user", "content": SP + "\n\n" + question}]
+    try:
+        resp = await client.chat.completions.create(
+            model="outputs/models/c18-2-combined-rs",
+            messages=messages, temperature=0.8, max_tokens=2048, n=n
+        )
+        return [c.message.content for c in resp.choices]
+    except Exception as e:
+        print(f"  Error: {e}")
+        return []
+
+async def main():
+    sem = asyncio.Semaphore(100)
+    sft_data = []
+    dpo_data = []
+    batch_size = 200
+
+    for batch_start in range(0, len(questions), batch_size):
+        batch = questions[batch_start:batch_start+batch_size]
+
+        async def process(q):
+            async with sem:
+                return q, await sample_one(q)
+
+        results = await asyncio.gather(*[process(q) for q in batch])
+
+        for q, answers in results:
+            gold = q_to_gold[q]
+            correct = []
+            incorrect = []
+            for a in answers:
+                pred = normalize(extract_boxed(a))
+                if pred and pred == gold:
+                    correct.append(a)
+                else:
+                    incorrect.append(a)
+
+            if correct:
+                best = min(correct, key=len)  # shortest correct
+                sft_data.append({
+                    "question": q, "answer": best,
+                    "n_correct": len(correct), "n_total": len(answers)
+                })
+
+            if correct and incorrect:
+                dpo_data.append({
+                    "question": q,
+                    "answer": min(correct, key=len),
+                    "bad_answer": max(incorrect, key=len)
+                })
+
+        print(f"  Batch {batch_start//batch_size + 1}: {len(sft_data)} sft, {len(dpo_data)} dpo")
+
+    import os
+    os.makedirs("outputs/c18_rs", exist_ok=True)
+    with open("outputs/c18_rs/sft_dataset.json", "w") as f:
+        json.dump(sft_data, f, ensure_ascii=False, indent=2)
+    with open("outputs/c18_rs/dpo_dataset.json", "w") as f:
+        json.dump(dpo_data, f, ensure_ascii=False, indent=2)
+
+    n4 = sum(1 for d in sft_data if d["n_correct"] >= 4)
+    print(f"\nRS Summary:")
+    print(f"  SFT: {len(sft_data)} (4+/16 filter: {n4})")
+    print(f"  DPO: {len(dpo_data)} pairs")
+    print(f"  Avg correct: {sum(d['n_correct'] for d in sft_data)/len(sft_data):.1f}/16")
+
+asyncio.run(main())
--- a/special_tokens_map.json
+++ b/special_tokens_map.json
@@ -0,0 +1,27 @@
+{
+  "boi_token": "<start_of_image>",
+  "bos_token": {
+    "content": "<bos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eoi_token": "<end_of_image>",
+  "eos_token": {
+    "content": "<eos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "image_token": "<image_soft_token>",
+  "pad_token": "<eos>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}
--- a/tokenizer.json
+++ b/tokenizer.json
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4667f2089529e8e7657cfb6d1c19910ae71ff5f28aa7ab2ff2763330affad795
+size 33384568
--- a/tokenizer.model
+++ b/tokenizer.model
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1299c11d7cf632ef3b4e11937501358ada021bbdf7c47638d13c0ee982f2e79c
+size 4689074
--- a/tokenizer_config.json
+++ b/tokenizer_config.json
--- a/train_grpo.py
+++ b/train_grpo.py
@@ -0,0 +1,171 @@
+"""C27: GRPO (Group Relative Policy Optimization) for math reasoning
+Based on DeepSeekMath GRPO + Gemma-2-2B success recipe from literature.
+"""
+import json, re, random, torch, numpy as np, os
+from datasets import Dataset
+from transformers import AutoTokenizer, AutoModelForCausalLM
+from trl import GRPOConfig, GRPOTrainer
+
+SEED = 42
+random.seed(SEED); np.random.seed(SEED); torch.manual_seed(SEED)
+if torch.cuda.is_available():
+    torch.cuda.manual_seed_all(SEED)
+    if torch.cuda.get_device_capability()[0] >= 8:
+        torch.set_float32_matmul_precision('high')
+
+BASE_MODEL = "outputs/models/c20-2-5x-replay"
+
+SP = "주어진 수학 문제를 단계별로 풀고 답변을 작성하세요.\n반드시 최종 답변을 \\boxed{정수} 형식으로 마지막 줄에 출력하세요.\n예시: \\boxed{42}"
+
+# === Load questions + ground truth ===
+with open("data/GSM8K_full_qwen3_30b.json") as f:
+    teacher_data = json.load(f)
+
+def extract_boxed(text):
+    m = re.findall(r'\\boxed\{([^}]+)\}', text)
+    return m[-1].strip() if m else None
+
+def normalize(a):
+    if a is None: return None
+    s = str(a).replace(",","").replace(" ","").strip()
+    try:
+        n = float(s)
+        return str(int(n)) if n == int(n) else str(n)
+    except: return s
+
+# Build ground truth
+gt_by_q = {}
+for t in teacher_data:
+    ans = extract_boxed(t["answer"])
+    if ans is None: continue
+    q = t["question"]
+    na = normalize(ans)
+    if q not in gt_by_q: gt_by_q[q] = {}
+    gt_by_q[q][na] = gt_by_q[q].get(na, 0) + 1
+
+ground_truth = {q: max(counts, key=counts.get) for q, counts in gt_by_q.items()}
+questions = list(ground_truth.keys())
+random.shuffle(questions)
+print(f"Total questions: {len(questions)}")
+
+# Build dataset with prompt (conversational format) + answer column
+dataset_items = []
+for q in questions:
+    dataset_items.append({
+        "prompt": [{"role": "user", "content": SP + "\n\n" + q}],
+        "answer": ground_truth[q],
+    })
+
+train_ds = Dataset.from_list(dataset_items)
+print(f"Train dataset: {len(train_ds)}")
+
+# === Reward function ===
+def math_reward(prompts, completions, answer, **kwargs):
+    """Reward: 1.0 if boxed answer matches ground truth, 0.0 otherwise.
+    Also +0.1 bonus for having boxed format (encourages format compliance)."""
+    rewards = []
+    for completion, gt in zip(completions, answer):
+        # Handle conversational format (list of dicts) or plain string
+        if isinstance(completion, list):
+            text = completion[-1]["content"] if completion else ""
+        elif isinstance(completion, dict):
+            text = completion.get("content", "")
+        else:
+            text = str(completion)
+
+        pred = extract_boxed(text)
+        if pred is None:
+            rewards.append(0.0)  # No boxed = 0
+        elif normalize(pred) == gt:
+            rewards.append(1.0)  # Correct = 1.0
+        else:
+            rewards.append(0.1)  # Wrong but has boxed format = 0.1
+    return rewards
+
+# === GRPO Config ===
+# Literature-informed settings for 1B model
+NUM_GEN = 8
+BATCH_SIZE = 8  # per device, must be divisible by num_generations
+GRAD_ACCUM = 4  # effective batch = 8 * 4 = 32, / 8 gen = 4 prompts per step
+
+config = GRPOConfig(
+    output_dir="outputs/c27_grpo_ckpt",
+    report_to="none",
+    seed=SEED,
+
+    # Generation
+    num_generations=NUM_GEN,
+    max_completion_length=1024,
+    temperature=0.7,
+
+    # GRPO algorithm
+    beta=0.04,           # Higher KL penalty to preserve format/quality
+    loss_type="grpo",    # Standard GRPO
+    epsilon=0.2,         # PPO-style clipping
+    scale_rewards="group",  # Normalize within group
+
+    # Training
+    num_train_epochs=1,
+    per_device_train_batch_size=BATCH_SIZE,
+    gradient_accumulation_steps=GRAD_ACCUM,
+    learning_rate=5e-6,
+    lr_scheduler_type="cosine",
+    warmup_ratio=0.05,
+    max_grad_norm=0.1,   # Very strict gradient clipping (from literature)
+    optim="paged_adamw_8bit",
+    bf16=True,
+    gradient_checkpointing=True,
+
+    # Logging & saving
+    logging_steps=10,
+    save_strategy="no",
+    max_steps=500,
+
+    # vLLM for fast generation
+    use_vllm=True,
+    vllm_mode="colocate",
+    vllm_gpu_memory_utilization=0.3,
+)
+
+print(f"Config: num_gen={NUM_GEN}, batch={BATCH_SIZE}, ga={GRAD_ACCUM}")
+print(f"  effective_batch={BATCH_SIZE * GRAD_ACCUM}, prompts_per_step={BATCH_SIZE * GRAD_ACCUM // NUM_GEN}")
+print(f"  lr={config.learning_rate}, beta={config.beta}, max_steps={config.max_steps}")
+print(f"  vllm={config.use_vllm}, mode={config.vllm_mode}")
+
+# === Load model ===
+print("\nLoading model...")
+model = AutoModelForCausalLM.from_pretrained(
+    BASE_MODEL, torch_dtype=torch.bfloat16,
+    attn_implementation="flash_attention_2",
+)
+tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
+tokenizer.pad_token = tokenizer.eos_token
+
+# === Train ===
+trainer = GRPOTrainer(
+    model=model,
+    reward_funcs=math_reward,
+    args=config,
+    train_dataset=train_ds,
+    processing_class=tokenizer,
+)
+
+print("\n=== Starting GRPO Training ===")
+result = trainer.train()
+print(f"\nTraining loss: {result.training_loss:.4f}")
+
+# Save
+SAVE_DIR = "outputs/models/c27-grpo"
+os.makedirs(SAVE_DIR, exist_ok=True)
+trainer.save_model(SAVE_DIR)
+tokenizer.save_pretrained(SAVE_DIR)
+print(f"Saved: {SAVE_DIR}")
+
+# Print training metrics summary
+logs = trainer.state.log_history
+reward_logs = [l for l in logs if "reward" in str(l)]
+for l in reward_logs[-5:]:
+    print(f"  step={l.get('step','?')}: reward={l.get('reward', l.get('rewards/mean','?'))}, "
+          f"completion_length={l.get('completion_length', '?')}")
+
+print("\n=== GRPO Training Complete ===")
--- a/train_rs_sft.py
+++ b/train_rs_sft.py
@@ -0,0 +1,143 @@
+"""C20: Variants of C18-2 (the 48.5% recipe) with different replay ratios"""
+import json, re, random, torch, numpy as np, os
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from trl import SFTTrainer, SFTConfig
+from datasets import Dataset
+
+SEED = 42
+random.seed(SEED); np.random.seed(SEED); torch.manual_seed(SEED)
+if torch.cuda.is_available():
+    torch.cuda.manual_seed_all(SEED)
+    if torch.cuda.get_device_capability()[0] >= 8: torch.set_float32_matmul_precision('high')
+
+SP = "주어진 수학 문제를 단계별로 풀고 답변을 작성하세요.\n반드시 최종 답변을 \\boxed{정수} 형식으로 마지막 줄에 출력하세요.\n예시: \\boxed{42}"
+BASE = "outputs/models/c17d-gemma-3-1b-it-Math"
+
+# Load RS1+RS2 (the winning combo)
+with open("outputs/c17d_rs/sft_dataset.json") as f:
+    rs1 = json.load(f)
+with open("outputs/c17d_rs2/sft_dataset.json") as f:
+    rs2 = json.load(f)
+
+seen = set()
+rs_combined = []
+for d in rs1 + rs2:
+    if d["n_correct"] < 4: continue
+    key = (d["question"], d["answer"])
+    if key not in seen:
+        seen.add(key)
+        rs_combined.append({"question": d["question"], "answer": d["answer"], "source": "gsm8k"})
+print(f"RS1+RS2 combined: {len(rs_combined)}")
+
+with open("data/GSM8K_full_qwen3_30b.json") as f:
+    orig_data = json.load(f)
+orig_filtered = [d for d in orig_data if len(d["answer"]) <= 1500]
+
+def to_sft(ex):
+    return {"prompt": [{"role":"user","content":SP+"\n\n"+ex["question"]}],
+            "completion": [{"role":"assistant","content":ex["answer"]}]}
+
+# === Condition 1: RS1+RS2 + 2x replay (more aggressive RS) ===
+print("\n=== C20-1: RS1+RS2 + 2x replay ===")
+random.seed(SEED)
+rs_qs = set(d["question"] for d in rs_combined)
+replay = [d for d in orig_filtered if d["question"] not in rs_qs]
+random.shuffle(replay)
+replay1 = replay[:int(len(rs_combined) * 2)]
+mixed1 = rs_combined + replay1
+random.shuffle(mixed1)
+print(f"  RS: {len(rs_combined)} + replay: {len(replay1)} = {len(mixed1)}")
+
+ds1 = Dataset.from_list(mixed1)
+cols = [c for c in ds1.column_names if c not in ["prompt","completion"]]
+ds1 = ds1.map(to_sft, remove_columns=cols)
+
+tokenizer = AutoTokenizer.from_pretrained(BASE)
+model = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.bfloat16, device_map="auto", attn_implementation='flash_attention_2')
+tokenizer.pad_token = tokenizer.eos_token
+model.gradient_checkpointing_enable(); model.config.use_cache = False
+
+cfg1 = SFTConfig(report_to='none', seed=SEED, num_train_epochs=1, warmup_ratio=0.05,
+    weight_decay=0.01, max_grad_norm=1.0, per_device_train_batch_size=8,
+    gradient_accumulation_steps=4, max_length=2048, lr_scheduler_type='cosine',
+    learning_rate=2e-6, bf16=True, optim="paged_adamw_8bit",
+    output_dir="outputs/c20_1_ckpt", logging_steps=25, save_strategy="no")
+trainer = SFTTrainer(model=model, processing_class=tokenizer, train_dataset=ds1, args=cfg1)
+r = trainer.train()
+print(f"  Loss: {r.training_loss:.4f}")
+
+SAVE1 = "outputs/models/c20-1-2x-replay"
+os.makedirs(SAVE1, exist_ok=True)
+model.eval(); model.save_pretrained(SAVE1, safe_serialization=False)
+tokenizer.save_pretrained(SAVE1)
+del model, trainer; torch.cuda.empty_cache()
+
+# === Condition 2: RS1+RS2 + 5x replay (more teacher data) ===
+print("\n=== C20-2: RS1+RS2 + 5x replay ===")
+random.seed(SEED)
+replay = [d for d in orig_filtered if d["question"] not in rs_qs]
+random.shuffle(replay)
+replay2 = replay[:int(len(rs_combined) * 5)]
+mixed2 = rs_combined + replay2
+random.shuffle(mixed2)
+print(f"  RS: {len(rs_combined)} + replay: {len(replay2)} = {len(mixed2)}")
+
+ds2 = Dataset.from_list(mixed2)
+cols = [c for c in ds2.column_names if c not in ["prompt","completion"]]
+ds2 = ds2.map(to_sft, remove_columns=cols)
+
+tokenizer = AutoTokenizer.from_pretrained(BASE)
+model = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.bfloat16, device_map="auto", attn_implementation='flash_attention_2')
+tokenizer.pad_token = tokenizer.eos_token
+model.gradient_checkpointing_enable(); model.config.use_cache = False
+
+cfg2 = SFTConfig(report_to='none', seed=SEED, num_train_epochs=1, warmup_ratio=0.05,
+    weight_decay=0.01, max_grad_norm=1.0, per_device_train_batch_size=8,
+    gradient_accumulation_steps=4, max_length=2048, lr_scheduler_type='cosine',
+    learning_rate=2e-6, bf16=True, optim="paged_adamw_8bit",
+    output_dir="outputs/c20_2_ckpt", logging_steps=25, save_strategy="no")
+trainer = SFTTrainer(model=model, processing_class=tokenizer, train_dataset=ds2, args=cfg2)
+r = trainer.train()
+print(f"  Loss: {r.training_loss:.4f}")
+
+SAVE2 = "outputs/models/c20-2-5x-replay"
+os.makedirs(SAVE2, exist_ok=True)
+model.eval(); model.save_pretrained(SAVE2, safe_serialization=False)
+tokenizer.save_pretrained(SAVE2)
+del model, trainer; torch.cuda.empty_cache()
+
+# === Condition 3: RS1+RS2 + 3x replay + lr=3e-6 (higher lr) ===
+print("\n=== C20-3: RS1+RS2 + 3x replay + lr=3e-6 ===")
+random.seed(SEED)
+replay = [d for d in orig_filtered if d["question"] not in rs_qs]
+random.shuffle(replay)
+replay3 = replay[:int(len(rs_combined) * 3)]
+mixed3 = rs_combined + replay3
+random.shuffle(mixed3)
+print(f"  RS: {len(rs_combined)} + replay: {len(replay3)} = {len(mixed3)}")
+
+ds3 = Dataset.from_list(mixed3)
+cols = [c for c in ds3.column_names if c not in ["prompt","completion"]]
+ds3 = ds3.map(to_sft, remove_columns=cols)
+
+tokenizer = AutoTokenizer.from_pretrained(BASE)
+model = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.bfloat16, device_map="auto", attn_implementation='flash_attention_2')
+tokenizer.pad_token = tokenizer.eos_token
+model.gradient_checkpointing_enable(); model.config.use_cache = False
+
+cfg3 = SFTConfig(report_to='none', seed=SEED, num_train_epochs=1, warmup_ratio=0.05,
+    weight_decay=0.01, max_grad_norm=1.0, per_device_train_batch_size=8,
+    gradient_accumulation_steps=4, max_length=2048, lr_scheduler_type='cosine',
+    learning_rate=3e-6, bf16=True, optim="paged_adamw_8bit",
+    output_dir="outputs/c20_3_ckpt", logging_steps=25, save_strategy="no")
+trainer = SFTTrainer(model=model, processing_class=tokenizer, train_dataset=ds3, args=cfg3)
+r = trainer.train()
+print(f"  Loss: {r.training_loss:.4f}")
+
+SAVE3 = "outputs/models/c20-3-lr3e-6"
+os.makedirs(SAVE3, exist_ok=True)
+model.eval(); model.save_pretrained(SAVE3, safe_serialization=False)
+tokenizer.save_pretrained(SAVE3)
+del model, trainer; torch.cuda.empty_cache()
+
+print("\n=== All conditions complete ===")
--- a/train_sft.py
+++ b/train_sft.py
@@ -0,0 +1,67 @@
+"""C17d: 모든 풀이 + 길이 필터 (1500자 이하만) + NaN 방지"""
+import json, re, random, torch, numpy as np, os
+from collections import defaultdict
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from trl import SFTTrainer, SFTConfig
+from transformers import EarlyStoppingCallback
+from datasets import Dataset
+
+SEED = 42
+random.seed(SEED); np.random.seed(SEED); torch.manual_seed(SEED)
+if torch.cuda.is_available():
+    torch.cuda.manual_seed_all(SEED)
+    if torch.cuda.get_device_capability()[0] >= 8: torch.set_float32_matmul_precision('high')
+
+SP = "주어진 수학 문제를 단계별로 풀고 답변을 작성하세요.\n반드시 최종 답변을 \\boxed{정수} 형식으로 마지막 줄에 출력하세요.\n예시: \\boxed{42}"
+
+print("=== C17d: All solutions, length-filtered (≤1500 chars) ===")
+
+with open("data/GSM8K_full_qwen3_30b.json") as f:
+    data = json.load(f)
+
+# 길이 필터: 1500자 이하만
+filtered = [d for d in data if len(d['answer']) <= 1500]
+print(f"원본: {len(data)}개 → 필터 후: {len(filtered)}개 (제거: {len(data)-len(filtered)})")
+
+random.shuffle(filtered)
+uq = len(set(d["question"] for d in filtered))
+print(f"Unique: {uq}, avg {len(filtered)/uq:.1f}/q")
+
+split = int(len(filtered) * 0.95)
+train, test = filtered[:split], filtered[split:]
+def to_sft(ex):
+    return {"prompt": [{"role":"user","content":SP+"\n\n"+ex["question"]}],
+            "completion": [{"role":"assistant","content":ex["answer"]}]}
+
+cols = [c for c in Dataset.from_list(train[:1]).column_names if c not in ["prompt","completion"]]
+train_ds = Dataset.from_list(train).map(to_sft, remove_columns=cols)
+test_ds = Dataset.from_list(test).map(to_sft, remove_columns=cols)
+print(f"학습: {len(train_ds)} / 검증: {len(test_ds)}")
+
+tokenizer = AutoTokenizer.from_pretrained("outputs/models/gemma-3-1b-it")
+model = AutoModelForCausalLM.from_pretrained("outputs/models/gemma-3-1b-it", dtype=torch.bfloat16, device_map="auto", attn_implementation='flash_attention_2')
+tokenizer.pad_token = tokenizer.eos_token
+model.gradient_checkpointing_enable(); model.config.use_cache = False
+
+cfg = SFTConfig(
+    report_to='none', seed=SEED, eval_strategy="steps", eval_steps=200,
+    save_total_limit=2, load_best_model_at_end=True, metric_for_best_model="eval_loss",
+    save_steps=200, num_train_epochs=3, warmup_ratio=0.05, weight_decay=0.01, max_grad_norm=1.0,
+    neftune_noise_alpha=5, per_device_train_batch_size=8, gradient_accumulation_steps=4,
+    per_device_eval_batch_size=2, max_length=2048, lr_scheduler_type='cosine',
+    learning_rate=2e-5, bf16=True, optim="paged_adamw_8bit",
+    output_dir="outputs/c17d_checkpoints", logging_steps=50, save_strategy="steps",
+)
+
+trainer = SFTTrainer(model=model, processing_class=tokenizer, train_dataset=train_ds, eval_dataset=test_ds, args=cfg,
+                     callbacks=[EarlyStoppingCallback(early_stopping_patience=3)])
+print("학습 시작 (3 epochs, 모든 풀이, ≤1500자)")
+r = trainer.train()
+print(f"완료! Loss: {r.training_loss:.4f}")
+
+SAVE = "outputs/models/c17d-gemma-3-1b-it-Math"
+os.makedirs(SAVE, exist_ok=True)
+model.eval(); model.save_pretrained(SAVE, safe_serialization=False); tokenizer.save_pretrained(SAVE)
+print(f"저장: {SAVE}")
+del model, trainer; torch.cuda.empty_cache()
+print("GPU 해제")
--- a/training_args.bin
+++ b/training_args.bin
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f40c5b3f028af33b116fcd2ab2d9a41be2525ca3e7de9916546dbd8c70e7d870
+size 7633