72 lines
2.7 KiB
Markdown
72 lines
2.7 KiB
Markdown
|
---
|
|||
|
|
license: apache-2.0
|
|||
|
|
base_model: Qwen/Qwen3-8B-Base
|
|||
|
|
library_name: transformers
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
tags:
|
|||
|
|
- qwen3
|
|||
|
|
- on-policy-distillation
|
|||
|
|
- eopd
|
|||
|
|
- search
|
|||
|
|
- tool-use
|
|||
|
|
- agent
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# Qwen3-8B-Base-Math-SeaSFT-Search-EOPD-Tau
|
|||
|
|
|
|||
|
|
An 8B agentic model trained with **Entropy-Aware On-Policy Distillation (EOPD)** in a
|
|||
|
|
multi-stage RL pipeline, built on **Qwen3-8B-Base**. This checkpoint is the final iteration
|
|||
|
|
(`iter_0000300`) of the Search-domain EOPD run.
|
|||
|
|
|
|||
|
|
## Training pipeline
|
|||
|
|
|
|||
|
|
| Stage | Description |
|
|||
|
|
|-------|-------------|
|
|||
|
|
| Base | `Qwen/Qwen3-8B-Base` |
|
|||
|
|
| SFT | Math + Search cold-start SFT, then Tau (tool-agent) SFT — student init `Qwen3-8B-Base-Math-SeaSFT-Search-TauSFT-Tau` |
|
|||
|
|
| EOPD | On-policy distillation from a **Search specialist teacher** (`Qwen3-8B-Base-Math-SeaSFT-Search`) |
|
|||
|
|
|
|||
|
|
## Method: Entropy-Aware On-Policy Distillation (EOPD)
|
|||
|
|
|
|||
|
|
EOPD combines two KL directions to get the best of mode-seeking and mode-covering distillation:
|
|||
|
|
|
|||
|
|
- **Reverse-KL OPD** everywhere (mode-seeking): the sampled-token log-ratio
|
|||
|
|
`student_logp − teacher_logp` is subtracted from the advantage, sharpening the student
|
|||
|
|
toward the teacher's modes.
|
|||
|
|
- **Forward-KL on high-entropy teacher tokens** (mode-covering): where reverse KL alone
|
|||
|
|
collapses diversity, a differentiable forward-KL loss
|
|||
|
|
`KL(p̃_teacher_topk ‖ p̃_student_topk)` is added, gated by the teacher's per-token entropy
|
|||
|
|
(`1[H_teacher > τ]`) and computed over the teacher's renormalized top-k (k=16) distribution.
|
|||
|
|
|
|||
|
|
The teacher is served with SGLang and returns both the scalar sampled-token logprob (reverse
|
|||
|
|
KL) and the per-token top-k distribution + entropy (forward KL). Training was done with the
|
|||
|
|
[slime](https://github.com/THUDM/slime) RL framework on Megatron-LM; this repo is the
|
|||
|
|
Hugging Face `safetensors` export of the final checkpoint.
|
|||
|
|
|
|||
|
|
## Architecture
|
|||
|
|
|
|||
|
|
Identical to Qwen3-8B-Base: 36 layers, hidden size 4096, 32 attention heads, 8 KV heads (GQA),
|
|||
|
|
head dim 128, QK-LayerNorm, RMSNorm, SwiGLU, RoPE (θ=1e6), 151,936 vocab, 32,768 context.
|
|||
|
|
|
|||
|
|
## Usage
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|||
|
|
|
|||
|
|
model_id = "willhx/Qwen3-8B-Base-Math-SeaSFT-Search-EOPD-Tau"
|
|||
|
|
tok = AutoTokenizer.from_pretrained(model_id)
|
|||
|
|
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16", device_map="auto")
|
|||
|
|
|
|||
|
|
prompt = "Who won the Nobel Prize in Physics in 1921?"
|
|||
|
|
inputs = tok(prompt, return_tensors="pt").to(model.device)
|
|||
|
|
out = model.generate(**inputs, max_new_tokens=256)
|
|||
|
|
print(tok.decode(out[0], skip_special_tokens=True))
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
The model is trained for a search/tool-use agent loop (Search-R1 style retrieval); to reproduce
|
|||
|
|
the agentic behavior, drive it with the same tool prompt/format used during training.
|
|||
|
|
|
|||
|
|
## License
|
|||
|
|
|
|||
|
|
Apache-2.0, inherited from Qwen3-8B-Base.
|