ModelHub XC ff3c87cd78 初始化项目,由ModelHub XC社区提供模型
Model: MMOPD/Qwen3-4B-OT3-law
Source: Original Platform
2026-11-14 04:46:18 +08:00

license, base_model, language, pipeline_tag, library_name, datasets, tags
license base_model language pipeline_tag library_name datasets tags
apache-2.0 Qwen/Qwen3-4B-Base
en
text-generation transformers
MMOPD/domain-sft-data
qwen3
reasoning
thinking
law

Qwen3-4B-OT3-law

The Law domain teacher of No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation (code).

Training

Initialization Qwen3-4B-OT3 (Qwen3-4B-Base after SFT on OpenThoughts3-1.2M)
Method SFT
Data law subset of MMOPD/domain-sft-data: 56,123 correct, deduplicated samples generated by Qwen3.6-35B-A3B from CaseHOLD and bar-exam prompts
Recipe 2 epochs, batch 128 packed sequences of 20,480 tokens, peak LR 1e-5, cosine decay with 5% warm-up, no weight decay, gradient clipping 1.0

Evaluation

CaseHOLD accuracy 73.3 (avg@3: three samples per prompt with seeds 42-44, temperature 1.0, top-p 1.0, 26,624-token completion cap).

Format

Qwen3 chat template in thinking mode. The reasoning follows the OpenThoughts3 format: the assistant turn starts with <think> (one space) and the reasoning, then \n</think>\n\n and the answer.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "MMOPD/Qwen3-4B-OT3-law"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype="auto", device_map="auto")
messages = [{"role": "user", "content": "..."}]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=True)
out = model.generate(**tok(text, return_tensors="pt").to(model.device), max_new_tokens=32768,
                     do_sample=True, temperature=1.0, top_p=1.0)
print(tok.decode(out[0], skip_special_tokens=True))

Citation

@article{kim2026immopd,
  title   = {No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation},
  author  = {Kim, Seonghyeon and Jang, Chaeyun and Lee, Noah and Kim, Boseop and Lee, Juho},
  journal = {arXiv preprint arXiv:2609.34745},
  year    = {2026}
}
S
Description
Model synced from source: MMOPD/Qwen3-4B-OT3-law
Readme
13 MiB
0 Stars 1 Watchers 0 Forks
Languages
Jinja 100%