1.7 KiB
1.7 KiB
license, base_model, tags, datasets, language, library_name
| license | base_model | tags | datasets | language | library_name | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| apache-2.0 | Qwen/Qwen2.5-7B-Instruct |
|
|
|
transformers |
Oracle internal-trace distillation · Qwen3-32B teacher -> Qwen2.5-7B-Instruct student
This repo holds the oracle internal-trace student for the 32B row of the REP
paper's main distillation comparison table (tab:maincomparison).
The student is Qwen/Qwen2.5-7B-Instruct full-fine-tuned (s1 recipe) on
Qwen3-32B's own hidden reasoning traces (<think> internal chain of thought,
no attack) sampled on 10k OpenThoughts prompts
(Chia-Mu-Lab/ot-ideal-q3_32b-clean, r1+boxed target).
The weights at the repo root are checkpoint-2015 (epoch 5), the checkpoint reported in the paper.
Reported metrics (this run, kod-s1-ot-ideal-q3_32b-clean-v1)
| ckpt | MATH500 | AIME24 | AIME25 | JEE (strict, full) | LCB pass@1 |
|---|---|---|---|---|---|
| base | 70.9 | 10.0 | 4.4 | 29.2 | 18.3 |
| step-01613 (ep4) | 67.6 | 12.2 | 13.3 | 32.9 | 17.6 |
| step-02015 (ep5) | 70.0 | 16.7 | 15.6 | 37.5 | 15.8 |
The paper table reports the JEE-Math subset (46.4 / 49.3 strict/partial) rather than the full-JEE strict number (37.5) shown here; all other cells match the paper row exactly (MATH500 70.0 / AIME24 16.7 / AIME25 15.6 / LCB 15.8).
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("Chia-Mu-Lab/qwen25-7b-ot-ideal-q3_32b-clean", torch_dtype="bfloat16")
tok = AutoTokenizer.from_pretrained("Chia-Mu-Lab/qwen25-7b-ot-ideal-q3_32b-clean")