main
Model: MMOPD/Qwen3-4B-OT3-law Source: Original Platform
license, base_model, language, pipeline_tag, library_name, datasets, tags
| license | base_model | language | pipeline_tag | library_name | datasets | tags | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| apache-2.0 | Qwen/Qwen3-4B-Base |
|
text-generation | transformers |
|
|
Qwen3-4B-OT3-law
The Law domain teacher of No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation (code).
Training
| Initialization | Qwen3-4B-OT3 (Qwen3-4B-Base after SFT on OpenThoughts3-1.2M) |
| Method | SFT |
| Data | law subset of MMOPD/domain-sft-data: 56,123 correct, deduplicated samples generated by Qwen3.6-35B-A3B from CaseHOLD and bar-exam prompts |
| Recipe | 2 epochs, batch 128 packed sequences of 20,480 tokens, peak LR 1e-5, cosine decay with 5% warm-up, no weight decay, gradient clipping 1.0 |
Evaluation
CaseHOLD accuracy 73.3 (avg@3: three samples per prompt with seeds 42-44, temperature 1.0, top-p 1.0, 26,624-token completion cap).
Format
Qwen3 chat template in thinking mode. The reasoning follows the OpenThoughts3 format: the assistant turn starts with <think> (one space) and the reasoning, then \n</think>\n\n and the answer.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "MMOPD/Qwen3-4B-OT3-law"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype="auto", device_map="auto")
messages = [{"role": "user", "content": "..."}]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=True)
out = model.generate(**tok(text, return_tensors="pt").to(model.device), max_new_tokens=32768,
do_sample=True, temperature=1.0, top_p=1.0)
print(tok.decode(out[0], skip_special_tokens=True))
Citation
@article{kim2026immopd,
title = {No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation},
author = {Kim, Seonghyeon and Jang, Chaeyun and Lee, Noah and Kim, Boseop and Lee, Juho},
journal = {arXiv preprint arXiv:2609.34745},
year = {2026}
}
Description
Model synced from source: MMOPD/Qwen3-4B-OT3-law
13 MiB
0 Stars
1 Watchers
0 Forks
Languages
Jinja
100%