72 lines
2.7 KiB
Markdown
72 lines
2.7 KiB
Markdown
---
|
||
license: apache-2.0
|
||
base_model: Qwen/Qwen3-8B-Base
|
||
library_name: transformers
|
||
pipeline_tag: text-generation
|
||
tags:
|
||
- qwen3
|
||
- on-policy-distillation
|
||
- eopd
|
||
- search
|
||
- tool-use
|
||
- agent
|
||
---
|
||
|
||
# Qwen3-8B-Base-Math-SeaSFT-Search-EOPD-Tau
|
||
|
||
An 8B agentic model trained with **Entropy-Aware On-Policy Distillation (EOPD)** in a
|
||
multi-stage RL pipeline, built on **Qwen3-8B-Base**. This checkpoint is the final iteration
|
||
(`iter_0000300`) of the Search-domain EOPD run.
|
||
|
||
## Training pipeline
|
||
|
||
| Stage | Description |
|
||
|-------|-------------|
|
||
| Base | `Qwen/Qwen3-8B-Base` |
|
||
| SFT | Math + Search cold-start SFT, then Tau (tool-agent) SFT — student init `Qwen3-8B-Base-Math-SeaSFT-Search-TauSFT-Tau` |
|
||
| EOPD | On-policy distillation from a **Search specialist teacher** (`Qwen3-8B-Base-Math-SeaSFT-Search`) |
|
||
|
||
## Method: Entropy-Aware On-Policy Distillation (EOPD)
|
||
|
||
EOPD combines two KL directions to get the best of mode-seeking and mode-covering distillation:
|
||
|
||
- **Reverse-KL OPD** everywhere (mode-seeking): the sampled-token log-ratio
|
||
`student_logp − teacher_logp` is subtracted from the advantage, sharpening the student
|
||
toward the teacher's modes.
|
||
- **Forward-KL on high-entropy teacher tokens** (mode-covering): where reverse KL alone
|
||
collapses diversity, a differentiable forward-KL loss
|
||
`KL(p̃_teacher_topk ‖ p̃_student_topk)` is added, gated by the teacher's per-token entropy
|
||
(`1[H_teacher > τ]`) and computed over the teacher's renormalized top-k (k=16) distribution.
|
||
|
||
The teacher is served with SGLang and returns both the scalar sampled-token logprob (reverse
|
||
KL) and the per-token top-k distribution + entropy (forward KL). Training was done with the
|
||
[slime](https://github.com/THUDM/slime) RL framework on Megatron-LM; this repo is the
|
||
Hugging Face `safetensors` export of the final checkpoint.
|
||
|
||
## Architecture
|
||
|
||
Identical to Qwen3-8B-Base: 36 layers, hidden size 4096, 32 attention heads, 8 KV heads (GQA),
|
||
head dim 128, QK-LayerNorm, RMSNorm, SwiGLU, RoPE (θ=1e6), 151,936 vocab, 32,768 context.
|
||
|
||
## Usage
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
||
model_id = "willhx/Qwen3-8B-Base-Math-SeaSFT-Search-EOPD-Tau"
|
||
tok = AutoTokenizer.from_pretrained(model_id)
|
||
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16", device_map="auto")
|
||
|
||
prompt = "Who won the Nobel Prize in Physics in 1921?"
|
||
inputs = tok(prompt, return_tensors="pt").to(model.device)
|
||
out = model.generate(**inputs, max_new_tokens=256)
|
||
print(tok.decode(out[0], skip_special_tokens=True))
|
||
```
|
||
|
||
The model is trained for a search/tool-use agent loop (Search-R1 style retrieval); to reproduce
|
||
the agentic behavior, drive it with the same tool prompt/format used during training.
|
||
|
||
## License
|
||
|
||
Apache-2.0, inherited from Qwen3-8B-Base.
|