68 lines
2.2 KiB
Markdown
68 lines
2.2 KiB
Markdown
|
---
|
|||
|
|
license: apache-2.0
|
|||
|
|
base_model: Qwen/Qwen3-0.6B
|
|||
|
|
datasets:
|
|||
|
|
- Divij/qwen3-32b-mas-traces
|
|||
|
|
language:
|
|||
|
|
- en
|
|||
|
|
library_name: transformers
|
|||
|
|
tags:
|
|||
|
|
- sft
|
|||
|
|
- qwen3
|
|||
|
|
- multi-agent
|
|||
|
|
- distillation
|
|||
|
|
- planner
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# STEVENZHANG904/Qwen3-0.6B-planner-sft
|
|||
|
|
|
|||
|
|
SFT-finetuned **Qwen/Qwen3-0.6B** on the **planner** subset of [Divij/qwen3-32b-mas-traces](https://huggingface.co/datasets/Divij/qwen3-32b-mas-traces),
|
|||
|
|
which contains traces of **Qwen3-32B** acting as a `planner` agent in a multi-agent system. This model is the
|
|||
|
|
distilled student that learns to play the same role as Qwen3-32B in that pipeline.
|
|||
|
|
|
|||
|
|
## Branches
|
|||
|
|
|
|||
|
|
| Branch | Epochs trained | Notes |
|
|||
|
|
|---|---|---|
|
|||
|
|
| `epoch2` | 2 | intermediate |
|
|||
|
|
| `epoch5` | 5 | intermediate |
|
|||
|
|
| `main` | 10 | final |
|
|||
|
|
|
|||
|
|
## Training configuration
|
|||
|
|
|
|||
|
|
- **Base model:** `Qwen/Qwen3-0.6B`
|
|||
|
|
- **Dataset:** `Divij/qwen3-32b-mas-traces` (config `planner`)
|
|||
|
|
- **Loss:** assistant-only (system + user tokens masked)
|
|||
|
|
- **Optimizer:** AdamW (β=(0.9, 0.95), wd=0.01, eps=1e-8)
|
|||
|
|
- **Learning rate:** 1e-5, constant with 3% warmup
|
|||
|
|
- **Sequence length:** 8192 (sequence packing on)
|
|||
|
|
- **Precision:** bf16
|
|||
|
|
- **Hardware:** 8× H100 80GB, DDP
|
|||
|
|
- **Liger-Kernel:** on (chunked CE + fused RMSNorm)
|
|||
|
|
|
|||
|
|
## Inference
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|||
|
|
import torch
|
|||
|
|
|
|||
|
|
repo = "STEVENZHANG904/Qwen3-0.6B-planner-sft"
|
|||
|
|
tok = AutoTokenizer.from_pretrained(repo)
|
|||
|
|
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="cuda")
|
|||
|
|
|
|||
|
|
# Planner role expects a task-spec prompt — see the dataset card for the exact format.
|
|||
|
|
messages = [
|
|||
|
|
{"role": "system", "content": "You are a helpful, creative, and smart assistant."},
|
|||
|
|
{"role": "user", "content": "<your planner task spec here>"},
|
|||
|
|
]
|
|||
|
|
inputs = tok.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True).to("cuda")
|
|||
|
|
out = model.generate(
|
|||
|
|
inputs, max_new_tokens=4096,
|
|||
|
|
do_sample=True, temperature=0.6, top_p=0.95, # Qwen3 thinking-mode defaults
|
|||
|
|
)
|
|||
|
|
print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
The model emits `<think>...</think>` reasoning blocks (inherited from Qwen3-32B traces).
|
|||
|
|
**Use sampling**, not greedy decoding — small distilled models can loop in `<think>` under greedy.
|