初始化项目，由ModelHub XC社区提供模型

Model: NotoriousH2/gemma-3-1b-it-Math-RS-SFT Source: Original Platform
2026-06-24 12:19:20 +08:00
commit 148c669d59
15 changed files with 52033 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text
--- a/README.md
+++ b/README.md
@@ -0,0 +1,84 @@
+---
+language: ko
+license: apache-2.0
+base_model: google/gemma-3-1b-it
+tags:
+- math
+- korean
+- rejection-sampling
+- sft
+- gemma
+datasets:
+- NotoriousH2/HRM8K
+---
+
+# Gemma-3-1B-IT Math RS-SFT (Best Model)
+
+SFT → Rejection Sampling → SFT 2단계 파이프라인으로 학습한 한국어 수학 모델. **최고 성능.**
+
+## 성능
+
+| Benchmark | Score |
+|-----------|-------|
+| HRM8K eval GSM8K (264문제, Korean) | **~46.6%** avg, **48.9%** best run |
+| HRM8K eval MATH (577문제, Korean) | ~17% |
+
+> ⚠️ temperature=0에서도 vLLM inference variance ±2-4%p 존재. 위 수치는 3회 평가 평균.
+
+## 데이터 생성 파이프라인
+
+### Stage 1: SFT 데이터 (교사 증류)
+위 SFT 모델과 동일. GSM8K 7,473문제 → Qwen3-30B로 한국어 풀이 26,254개 생성.
+
+### Stage 2: RS 데이터 (On-policy 샘플링)
+
+#### RS 샘플링
+
+
+#### RS 데이터 필터링
+
+
+#### RS-SFT 학습 데이터 구성 (핵심!)
+
+
+**Replay가 핵심**: RS 데이터만 사용하면 교사 풀이 패턴을 잊어 성능 하락 (catastrophic forgetting).
+
+| Replay 비율 | GSM8K | 비고 |
+|------------|-------|------|
+| 0x (RS only) | 46.2% | forgetting |
+| 2x | 46.6% | 부족 |
+| 3x | 48.5% | 양호 |
+| **5x** | **48.9%** | **최적** |
+| max (전부) | 47.3% | RS 희석 |
+
+### RS-SFT 학습 데이터 형식
+SFT와 동일한 question/answer JSON. 차이점은 answer가 학생 모델(SFT)이 스스로 생성한 정답 풀이라는 것.
+
+## 학습 설정
+
+### Stage 1: SFT
+
+
+### Stage 2: RS-SFT
+
+
+## 재현 방법
+
+INFO 03-19 14:53:13 [__init__.py:216] Automatically detected platform cuda.
+[1;36m(APIServer pid=3428638)[0;0m INFO 03-19 14:53:19 [api_server.py:1839] vLLM API server version 0.11.0
+[1;36m(APIServer pid=3428638)[0;0m INFO 03-19 14:53:19 [utils.py:233] non-default args: {'model_tag': './sft_model', 'model': './sft_model', 'dtype': 'bfloat16', 'max_model_len': 4096, 'gpu_memory_utilization': 0.85}
+INFO 03-19 14:53:25 [__init__.py:216] Automatically detected platform cuda.
+[1;36m(APIServer pid=3428911)[0;0m INFO 03-19 14:53:31 [api_server.py:1839] vLLM API server version 0.11.0
+[1;36m(APIServer pid=3428911)[0;0m INFO 03-19 14:53:31 [utils.py:233] non-default args: {'model_tag': './rs_sft_model', 'model': './rs_sft_model', 'dtype': 'bfloat16', 'max_model_len': 4096, 'gpu_memory_utilization': 0.85}
+
+## 실패한 접근들 (참고)
+- Iterative RS (RS 모델 위에 다시 RS): 항상 퇴보
+- DPO (10가지 시도): 모두 무효 (1B 모델 capacity 부족)
+- GRPO (2가지 시도): base variance 범위 내
+- 다른 교사 모델: 스타일 불일치로 대폭 하락
+
+## 파일
+- : Stage 1 SFT 학습
+- : RS 샘플링 스크립트 (vLLM 서빙 필요)
+- : Stage 2 RS-SFT 학습 (replay 포함)
+- : HRM8K 평가
--- a/added_tokens.json
+++ b/added_tokens.json
@@ -0,0 +1,3 @@
+{
+  "<image_soft_token>": 262144
+}
--- a/chat_template.jinja
+++ b/chat_template.jinja
@@ -0,0 +1,47 @@
+{{ bos_token }}
+{%- if messages[0]['role'] == 'system' -%}
+    {%- if messages[0]['content'] is string -%}
+        {%- set first_user_prefix = messages[0]['content'] + '
+
+' -%}
+    {%- else -%}
+        {%- set first_user_prefix = messages[0]['content'][0]['text'] + '
+
+' -%}
+    {%- endif -%}
+    {%- set loop_messages = messages[1:] -%}
+{%- else -%}
+    {%- set first_user_prefix = "" -%}
+    {%- set loop_messages = messages -%}
+{%- endif -%}
+{%- for message in loop_messages -%}
+    {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}
+        {{ raise_exception("Conversation roles must alternate user/assistant/user/assistant/...") }}
+    {%- endif -%}
+    {%- if (message['role'] == 'assistant') -%}
+        {%- set role = "model" -%}
+    {%- else -%}
+        {%- set role = message['role'] -%}
+    {%- endif -%}
+    {{ '<start_of_turn>' + role + '
+' + (first_user_prefix if loop.first else "") }}
+    {%- if message['content'] is string -%}
+        {{ message['content'] | trim }}
+    {%- elif message['content'] is iterable -%}
+        {%- for item in message['content'] -%}
+            {%- if item['type'] == 'image' -%}
+                {{ '<start_of_image>' }}
+            {%- elif item['type'] == 'text' -%}
+                {{ item['text'] | trim }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{ raise_exception("Invalid content type") }}
+    {%- endif -%}
+    {{ '<end_of_turn>
+' }}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    {{'<start_of_turn>model
+'}}
+{%- endif -%}
--- a/config.json
+++ b/config.json
@@ -0,0 +1,64 @@
+{
+  "_sliding_window_pattern": 6,
+  "architectures": [
+    "Gemma3ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "attn_logit_softcapping": null,
+  "bos_token_id": 2,
+  "cache_implementation": "hybrid",
+  "dtype": "bfloat16",
+  "eos_token_id": 1,
+  "final_logit_softcapping": null,
+  "head_dim": 256,
+  "hidden_activation": "gelu_pytorch_tanh",
+  "hidden_size": 1152,
+  "initializer_range": 0.02,
+  "intermediate_size": 6912,
+  "layer_types": [
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention"
+  ],
+  "max_position_embeddings": 32768,
+  "model_type": "gemma3_text",
+  "num_attention_heads": 4,
+  "num_hidden_layers": 26,
+  "num_key_value_heads": 1,
+  "pad_token_id": 1,
+  "query_pre_attn_scalar": 256,
+  "rms_norm_eps": 1e-06,
+  "rope_local_base_freq": 10000,
+  "rope_scaling": null,
+  "rope_theta": 1000000,
+  "sliding_window": 512,
+  "sliding_window_pattern": 6,
+  "transformers_version": "4.57.3",
+  "use_bidirectional_attention": false,
+  "use_cache": false,
+  "vocab_size": 262144
+}
--- a/eval.py
+++ b/eval.py
@@ -0,0 +1,86 @@
+"""공통 평가 스크립트: vLLM 서버에 연결하여 HRM8K 전체 841문제 평가 (temperature=0)"""
+import os, json, re, sys, asyncio
+from langchain_openai import ChatOpenAI
+from langchain_core.prompts import ChatPromptTemplate
+from langchain_core.output_parsers import StrOutputParser
+from openai import OpenAI
+
+MATH_SYSTEM_PROMPT = """주어진 수학 문제를 단계별로 풀고 답변을 작성하세요.
+반드시 최종 답변을 \\boxed{정수} 형식으로 마지막 줄에 출력하세요.
+예시: \\boxed{42}"""
+
+def extract_boxed(text):
+    m = re.findall(r'\\boxed\{([^}]+)\}', text)
+    return m[-1].strip() if m else None
+
+def normalize(a):
+    if a is None: return None
+    s = str(a).replace(",","").replace(" ","").strip()
+    try:
+        n = float(s)
+        return str(int(n)) if n == int(n) else str(n)
+    except: return s
+
+def check(pred, gt):
+    p, g = normalize(pred), normalize(gt)
+    return p is not None and g is not None and p == g
+
+async def evaluate(label="", save_path=None):
+    client = OpenAI(base_url="http://localhost:8000/v1", api_key="token-abc123")
+    model_name = client.models.list().data[0].id
+    print(f"모델: {model_name}")
+
+    with open("data/HRM8k_eval.json") as f:
+        data = json.load(f)
+    print(f"평가: {len(data)}개 (temperature=0, max_tokens=2048)")
+
+    llm = ChatOpenAI(base_url="http://localhost:8000/v1", api_key="token-abc123",
+                     model=model_name, temperature=0, max_tokens=2048)
+    prompt = ChatPromptTemplate([("user", "{sp}\n\n{q}")]).partial(sp=MATH_SYSTEM_PROMPT)
+    chain = prompt | llm | StrOutputParser()
+    inputs = [{"q": item["question"]} for item in data]
+    results = await chain.abatch(inputs, config={"max_concurrency": 400})
+
+    by_src = {}
+    details = []
+    for item, res in zip(data, results):
+        s = item.get("source", "?")
+        if s not in by_src: by_src[s] = {"correct": 0, "total": 0, "no_boxed": 0}
+        by_src[s]["total"] += 1
+        pred = extract_boxed(res)
+        is_correct = False
+        if pred is None:
+            by_src[s]["no_boxed"] += 1
+        elif check(pred, item["answer"]):
+            by_src[s]["correct"] += 1
+            is_correct = True
+        details.append({
+            "question": item["question"][:80],
+            "source": s,
+            "gt": str(item["answer"])[-30:] if isinstance(item["answer"], str) else str(item["answer"]),
+            "pred": pred,
+            "correct": is_correct,
+        })
+
+    tc = sum(v["correct"] for v in by_src.values())
+    tt = sum(v["total"] for v in by_src.values())
+    print(f"\n=== {label} 결과 (temperature=0) ===")
+    for s in sorted(by_src):
+        v = by_src[s]
+        print(f"  [{s.upper()}] {v['correct']}/{v['total']} ({v['correct']/v['total']*100:.1f}%) | boxed미출력: {v['no_boxed']}")
+    print(f"  [전체] {tc}/{tt} ({tc/tt*100:.1f}%)")
+
+    result_obj = {"label": label, "correct": tc, "total": tt, "accuracy": tc/tt*100, "by_source": by_src}
+
+    if save_path:
+        os.makedirs(os.path.dirname(save_path) or ".", exist_ok=True)
+        with open(save_path, "w") as f:
+            json.dump({"result": result_obj, "details": details}, f, ensure_ascii=False, indent=2)
+        print(f"  결과 저장: {save_path}")
+
+    return result_obj
+
+if __name__ == "__main__":
+    label = sys.argv[1] if len(sys.argv) > 1 else "eval"
+    save_path = sys.argv[2] if len(sys.argv) > 2 else None
+    asyncio.run(evaluate(label, save_path))
--- a/generation_config.json
+++ b/generation_config.json
@@ -0,0 +1,14 @@
+{
+  "bos_token_id": 2,
+  "cache_implementation": "hybrid",
+  "do_sample": true,
+  "eos_token_id": [
+    1,
+    1,
+    106
+  ],
+  "pad_token_id": 1,
+  "top_k": 64,
+  "top_p": 0.95,
+  "transformers_version": "4.57.3"
+}
--- a/pytorch_model.bin
+++ b/pytorch_model.bin
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2a9365c364f51eeae32a2916d48fc4377eb8c1c6bfd4f2ccc08231528d399178
+size 1999887347
--- a/rs_sample.py
+++ b/rs_sample.py
@@ -0,0 +1,108 @@
+"""RS sampling from C18-2 model (48.5% GSM8K)"""
+import json, re, asyncio, random
+from openai import AsyncOpenAI
+
+SP = "주어진 수학 문제를 단계별로 풀고 답변을 작성하세요.\n반드시 최종 답변을 \\boxed{정수} 형식으로 마지막 줄에 출력하세요.\n예시: \\boxed{42}"
+
+def extract_boxed(text):
+    m = re.findall(r'\\boxed\{([^}]+)\}', text)
+    return m[-1].strip() if m else None
+
+def normalize(a):
+    if a is None: return None
+    s = str(a).replace(",","").replace(" ","").strip()
+    try:
+        n = float(s)
+        return str(int(n)) if n == int(n) else str(n)
+    except: return s
+
+# Load GSM8K train questions (the ones we have gold answers for)
+with open("data/GSM8K_full_qwen3_30b.json") as f:
+    data = json.load(f)
+
+# Get unique questions with their gold answers
+q_to_gold = {}
+for d in data:
+    q = d["question"]
+    if q not in q_to_gold:
+        # Extract gold from the existing correct solutions
+        gold = extract_boxed(d["answer"])
+        if gold:
+            q_to_gold[q] = normalize(gold)
+
+questions = list(q_to_gold.keys())
+random.seed(42)
+random.shuffle(questions)
+# Sample a subset for RS (use all unique questions)
+print(f"Total unique questions: {len(questions)}")
+
+client = AsyncOpenAI(base_url="http://localhost:8000/v1", api_key="token-abc123")
+
+async def sample_one(question, n=16):
+    messages = [{"role": "user", "content": SP + "\n\n" + question}]
+    try:
+        resp = await client.chat.completions.create(
+            model="outputs/models/c18-2-combined-rs",
+            messages=messages, temperature=0.8, max_tokens=2048, n=n
+        )
+        return [c.message.content for c in resp.choices]
+    except Exception as e:
+        print(f"  Error: {e}")
+        return []
+
+async def main():
+    sem = asyncio.Semaphore(100)
+    sft_data = []
+    dpo_data = []
+    batch_size = 200
+
+    for batch_start in range(0, len(questions), batch_size):
+        batch = questions[batch_start:batch_start+batch_size]
+
+        async def process(q):
+            async with sem:
+                return q, await sample_one(q)
+
+        results = await asyncio.gather(*[process(q) for q in batch])
+
+        for q, answers in results:
+            gold = q_to_gold[q]
+            correct = []
+            incorrect = []
+            for a in answers:
+                pred = normalize(extract_boxed(a))
+                if pred and pred == gold:
+                    correct.append(a)
+                else:
+                    incorrect.append(a)
+
+            if correct:
+                best = min(correct, key=len)  # shortest correct
+                sft_data.append({
+                    "question": q, "answer": best,
+                    "n_correct": len(correct), "n_total": len(answers)
+                })
+
+            if correct and incorrect:
+                dpo_data.append({
+                    "question": q,
+                    "answer": min(correct, key=len),
+                    "bad_answer": max(incorrect, key=len)
+                })
+
+        print(f"  Batch {batch_start//batch_size + 1}: {len(sft_data)} sft, {len(dpo_data)} dpo")
+
+    import os
+    os.makedirs("outputs/c18_rs", exist_ok=True)
+    with open("outputs/c18_rs/sft_dataset.json", "w") as f:
+        json.dump(sft_data, f, ensure_ascii=False, indent=2)
+    with open("outputs/c18_rs/dpo_dataset.json", "w") as f:
+        json.dump(dpo_data, f, ensure_ascii=False, indent=2)
+
+    n4 = sum(1 for d in sft_data if d["n_correct"] >= 4)
+    print(f"\nRS Summary:")
+    print(f"  SFT: {len(sft_data)} (4+/16 filter: {n4})")
+    print(f"  DPO: {len(dpo_data)} pairs")
+    print(f"  Avg correct: {sum(d['n_correct'] for d in sft_data)/len(sft_data):.1f}/16")
+
+asyncio.run(main())
--- a/special_tokens_map.json
+++ b/special_tokens_map.json
@@ -0,0 +1,27 @@
+{
+  "boi_token": "<start_of_image>",
+  "bos_token": {
+    "content": "<bos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eoi_token": "<end_of_image>",
+  "eos_token": {
+    "content": "<eos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "image_token": "<image_soft_token>",
+  "pad_token": "<eos>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}
--- a/tokenizer.json
+++ b/tokenizer.json
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4667f2089529e8e7657cfb6d1c19910ae71ff5f28aa7ab2ff2763330affad795
+size 33384568
--- a/tokenizer.model
+++ b/tokenizer.model
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1299c11d7cf632ef3b4e11937501358ada021bbdf7c47638d13c0ee982f2e79c
+size 4689074
--- a/tokenizer_config.json
+++ b/tokenizer_config.json
--- a/train_rs_sft.py
+++ b/train_rs_sft.py
@@ -0,0 +1,143 @@
+"""C20: Variants of C18-2 (the 48.5% recipe) with different replay ratios"""
+import json, re, random, torch, numpy as np, os
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from trl import SFTTrainer, SFTConfig
+from datasets import Dataset
+
+SEED = 42
+random.seed(SEED); np.random.seed(SEED); torch.manual_seed(SEED)
+if torch.cuda.is_available():
+    torch.cuda.manual_seed_all(SEED)
+    if torch.cuda.get_device_capability()[0] >= 8: torch.set_float32_matmul_precision('high')
+
+SP = "주어진 수학 문제를 단계별로 풀고 답변을 작성하세요.\n반드시 최종 답변을 \\boxed{정수} 형식으로 마지막 줄에 출력하세요.\n예시: \\boxed{42}"
+BASE = "outputs/models/c17d-gemma-3-1b-it-Math"
+
+# Load RS1+RS2 (the winning combo)
+with open("outputs/c17d_rs/sft_dataset.json") as f:
+    rs1 = json.load(f)
+with open("outputs/c17d_rs2/sft_dataset.json") as f:
+    rs2 = json.load(f)
+
+seen = set()
+rs_combined = []
+for d in rs1 + rs2:
+    if d["n_correct"] < 4: continue
+    key = (d["question"], d["answer"])
+    if key not in seen:
+        seen.add(key)
+        rs_combined.append({"question": d["question"], "answer": d["answer"], "source": "gsm8k"})
+print(f"RS1+RS2 combined: {len(rs_combined)}")
+
+with open("data/GSM8K_full_qwen3_30b.json") as f:
+    orig_data = json.load(f)
+orig_filtered = [d for d in orig_data if len(d["answer"]) <= 1500]
+
+def to_sft(ex):
+    return {"prompt": [{"role":"user","content":SP+"\n\n"+ex["question"]}],
+            "completion": [{"role":"assistant","content":ex["answer"]}]}
+
+# === Condition 1: RS1+RS2 + 2x replay (more aggressive RS) ===
+print("\n=== C20-1: RS1+RS2 + 2x replay ===")
+random.seed(SEED)
+rs_qs = set(d["question"] for d in rs_combined)
+replay = [d for d in orig_filtered if d["question"] not in rs_qs]
+random.shuffle(replay)
+replay1 = replay[:int(len(rs_combined) * 2)]
+mixed1 = rs_combined + replay1
+random.shuffle(mixed1)
+print(f"  RS: {len(rs_combined)} + replay: {len(replay1)} = {len(mixed1)}")
+
+ds1 = Dataset.from_list(mixed1)
+cols = [c for c in ds1.column_names if c not in ["prompt","completion"]]
+ds1 = ds1.map(to_sft, remove_columns=cols)
+
+tokenizer = AutoTokenizer.from_pretrained(BASE)
+model = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.bfloat16, device_map="auto", attn_implementation='flash_attention_2')
+tokenizer.pad_token = tokenizer.eos_token
+model.gradient_checkpointing_enable(); model.config.use_cache = False
+
+cfg1 = SFTConfig(report_to='none', seed=SEED, num_train_epochs=1, warmup_ratio=0.05,
+    weight_decay=0.01, max_grad_norm=1.0, per_device_train_batch_size=8,
+    gradient_accumulation_steps=4, max_length=2048, lr_scheduler_type='cosine',
+    learning_rate=2e-6, bf16=True, optim="paged_adamw_8bit",
+    output_dir="outputs/c20_1_ckpt", logging_steps=25, save_strategy="no")
+trainer = SFTTrainer(model=model, processing_class=tokenizer, train_dataset=ds1, args=cfg1)
+r = trainer.train()
+print(f"  Loss: {r.training_loss:.4f}")
+
+SAVE1 = "outputs/models/c20-1-2x-replay"
+os.makedirs(SAVE1, exist_ok=True)
+model.eval(); model.save_pretrained(SAVE1, safe_serialization=False)
+tokenizer.save_pretrained(SAVE1)
+del model, trainer; torch.cuda.empty_cache()
+
+# === Condition 2: RS1+RS2 + 5x replay (more teacher data) ===
+print("\n=== C20-2: RS1+RS2 + 5x replay ===")
+random.seed(SEED)
+replay = [d for d in orig_filtered if d["question"] not in rs_qs]
+random.shuffle(replay)
+replay2 = replay[:int(len(rs_combined) * 5)]
+mixed2 = rs_combined + replay2
+random.shuffle(mixed2)
+print(f"  RS: {len(rs_combined)} + replay: {len(replay2)} = {len(mixed2)}")
+
+ds2 = Dataset.from_list(mixed2)
+cols = [c for c in ds2.column_names if c not in ["prompt","completion"]]
+ds2 = ds2.map(to_sft, remove_columns=cols)
+
+tokenizer = AutoTokenizer.from_pretrained(BASE)
+model = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.bfloat16, device_map="auto", attn_implementation='flash_attention_2')
+tokenizer.pad_token = tokenizer.eos_token
+model.gradient_checkpointing_enable(); model.config.use_cache = False
+
+cfg2 = SFTConfig(report_to='none', seed=SEED, num_train_epochs=1, warmup_ratio=0.05,
+    weight_decay=0.01, max_grad_norm=1.0, per_device_train_batch_size=8,
+    gradient_accumulation_steps=4, max_length=2048, lr_scheduler_type='cosine',
+    learning_rate=2e-6, bf16=True, optim="paged_adamw_8bit",
+    output_dir="outputs/c20_2_ckpt", logging_steps=25, save_strategy="no")
+trainer = SFTTrainer(model=model, processing_class=tokenizer, train_dataset=ds2, args=cfg2)
+r = trainer.train()
+print(f"  Loss: {r.training_loss:.4f}")
+
+SAVE2 = "outputs/models/c20-2-5x-replay"
+os.makedirs(SAVE2, exist_ok=True)
+model.eval(); model.save_pretrained(SAVE2, safe_serialization=False)
+tokenizer.save_pretrained(SAVE2)
+del model, trainer; torch.cuda.empty_cache()
+
+# === Condition 3: RS1+RS2 + 3x replay + lr=3e-6 (higher lr) ===
+print("\n=== C20-3: RS1+RS2 + 3x replay + lr=3e-6 ===")
+random.seed(SEED)
+replay = [d for d in orig_filtered if d["question"] not in rs_qs]
+random.shuffle(replay)
+replay3 = replay[:int(len(rs_combined) * 3)]
+mixed3 = rs_combined + replay3
+random.shuffle(mixed3)
+print(f"  RS: {len(rs_combined)} + replay: {len(replay3)} = {len(mixed3)}")
+
+ds3 = Dataset.from_list(mixed3)
+cols = [c for c in ds3.column_names if c not in ["prompt","completion"]]
+ds3 = ds3.map(to_sft, remove_columns=cols)
+
+tokenizer = AutoTokenizer.from_pretrained(BASE)
+model = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.bfloat16, device_map="auto", attn_implementation='flash_attention_2')
+tokenizer.pad_token = tokenizer.eos_token
+model.gradient_checkpointing_enable(); model.config.use_cache = False
+
+cfg3 = SFTConfig(report_to='none', seed=SEED, num_train_epochs=1, warmup_ratio=0.05,
+    weight_decay=0.01, max_grad_norm=1.0, per_device_train_batch_size=8,
+    gradient_accumulation_steps=4, max_length=2048, lr_scheduler_type='cosine',
+    learning_rate=3e-6, bf16=True, optim="paged_adamw_8bit",
+    output_dir="outputs/c20_3_ckpt", logging_steps=25, save_strategy="no")
+trainer = SFTTrainer(model=model, processing_class=tokenizer, train_dataset=ds3, args=cfg3)
+r = trainer.train()
+print(f"  Loss: {r.training_loss:.4f}")
+
+SAVE3 = "outputs/models/c20-3-lr3e-6"
+os.makedirs(SAVE3, exist_ok=True)
+model.eval(); model.save_pretrained(SAVE3, safe_serialization=False)
+tokenizer.save_pretrained(SAVE3)
+del model, trainer; torch.cuda.empty_cache()
+
+print("\n=== All conditions complete ===")
--- a/train_sft.py
+++ b/train_sft.py
@@ -0,0 +1,67 @@
+"""C17d: 모든 풀이 + 길이 필터 (1500자 이하만) + NaN 방지"""
+import json, re, random, torch, numpy as np, os
+from collections import defaultdict
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from trl import SFTTrainer, SFTConfig
+from transformers import EarlyStoppingCallback
+from datasets import Dataset
+
+SEED = 42
+random.seed(SEED); np.random.seed(SEED); torch.manual_seed(SEED)
+if torch.cuda.is_available():
+    torch.cuda.manual_seed_all(SEED)
+    if torch.cuda.get_device_capability()[0] >= 8: torch.set_float32_matmul_precision('high')
+
+SP = "주어진 수학 문제를 단계별로 풀고 답변을 작성하세요.\n반드시 최종 답변을 \\boxed{정수} 형식으로 마지막 줄에 출력하세요.\n예시: \\boxed{42}"
+
+print("=== C17d: All solutions, length-filtered (≤1500 chars) ===")
+
+with open("data/GSM8K_full_qwen3_30b.json") as f:
+    data = json.load(f)
+
+# 길이 필터: 1500자 이하만
+filtered = [d for d in data if len(d['answer']) <= 1500]
+print(f"원본: {len(data)}개 → 필터 후: {len(filtered)}개 (제거: {len(data)-len(filtered)})")
+
+random.shuffle(filtered)
+uq = len(set(d["question"] for d in filtered))
+print(f"Unique: {uq}, avg {len(filtered)/uq:.1f}/q")
+
+split = int(len(filtered) * 0.95)
+train, test = filtered[:split], filtered[split:]
+def to_sft(ex):
+    return {"prompt": [{"role":"user","content":SP+"\n\n"+ex["question"]}],
+            "completion": [{"role":"assistant","content":ex["answer"]}]}
+
+cols = [c for c in Dataset.from_list(train[:1]).column_names if c not in ["prompt","completion"]]
+train_ds = Dataset.from_list(train).map(to_sft, remove_columns=cols)
+test_ds = Dataset.from_list(test).map(to_sft, remove_columns=cols)
+print(f"학습: {len(train_ds)} / 검증: {len(test_ds)}")
+
+tokenizer = AutoTokenizer.from_pretrained("outputs/models/gemma-3-1b-it")
+model = AutoModelForCausalLM.from_pretrained("outputs/models/gemma-3-1b-it", dtype=torch.bfloat16, device_map="auto", attn_implementation='flash_attention_2')
+tokenizer.pad_token = tokenizer.eos_token
+model.gradient_checkpointing_enable(); model.config.use_cache = False
+
+cfg = SFTConfig(
+    report_to='none', seed=SEED, eval_strategy="steps", eval_steps=200,
+    save_total_limit=2, load_best_model_at_end=True, metric_for_best_model="eval_loss",
+    save_steps=200, num_train_epochs=3, warmup_ratio=0.05, weight_decay=0.01, max_grad_norm=1.0,
+    neftune_noise_alpha=5, per_device_train_batch_size=8, gradient_accumulation_steps=4,
+    per_device_eval_batch_size=2, max_length=2048, lr_scheduler_type='cosine',
+    learning_rate=2e-5, bf16=True, optim="paged_adamw_8bit",
+    output_dir="outputs/c17d_checkpoints", logging_steps=50, save_strategy="steps",
+)
+
+trainer = SFTTrainer(model=model, processing_class=tokenizer, train_dataset=train_ds, eval_dataset=test_ds, args=cfg,
+                     callbacks=[EarlyStoppingCallback(early_stopping_patience=3)])
+print("학습 시작 (3 epochs, 모든 풀이, ≤1500자)")
+r = trainer.train()
+print(f"완료! Loss: {r.training_loss:.4f}")
+
+SAVE = "outputs/models/c17d-gemma-3-1b-it-Math"
+os.makedirs(SAVE, exist_ok=True)
+model.eval(); model.save_pretrained(SAVE, safe_serialization=False); tokenizer.save_pretrained(SAVE)
+print(f"저장: {SAVE}")
+del model, trainer; torch.cuda.empty_cache()
+print("GPU 해제")