anurag203/clarify-rl-run4-qwen3-1.7b-beta0.2
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity
Files
main
clarify-rl-run4-qwen3-1.7b-…/README.md

294 lines
12 KiB
Markdown
Raw Permalink Normal View History

初始化项目，由ModelHub XC社区提供模型 Model: anurag203/clarify-rl-run4-qwen3-1.7b-beta0.2 Source: Original Platform
2026-06-16 05:46:18 +08:00
---
license: apache-2.0
base_model: Qwen/Qwen3-1.7B
base_model_relation: finetune
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- text-generation
- conversational
- reinforcement-learning
- rlhf
- rl
- grpo
- trl
- kl-anchor
- agentic
- tool-use
- clarifying-questions
- ask-vs-guess
model-index:
- name: clarify-rl-run4-qwen3-1.7b-beta0.2
results:
- task:
type: text-generation
name: ClarifyRL multi-turn ask-or-guess
dataset:
type: held_out_scenarios
name: ClarifyRL eval (50 held-out scenarios, n=50 v4)
metrics:
- type: avg_score
value: 0.0560
name: avg_score (μ over 4 families × 50 scenarios)
- type: completion_rate
value: 0.14
name: completion_rate
- type: family_event_planning_mean
value: 0.175
name: event_planning μ
- type: family_meeting_scheduling_mean
value: 0.064
name: meeting_scheduling μ
---
# ClarifyRL — Run 4 — Qwen3-1.7B GRPO (β=0.2 KL anchor)
> **The hero checkpoint of the ClarifyRL hackathon submission.** This is
> Qwen3-1.7B trained with TRL GRPO and an explicit **KL anchor at β=0.2**
> against the frozen base, which is the single change that turned a
> capability collapse (Run 2, β=0) into a measurable improvement on the
> held-out eval *while preserving breadth across families*.
- **Live demo (replay + CPU live chat):** [`anurag203/clarify-rl-demo`](https://huggingface.co/spaces/anurag203/clarify-rl-demo)
- **Code (training, eval, plots):** [`github.com/anurag203/clarify-rl`](https://github.com/anurag203/clarify-rl)
- **W&B dashboard (all 3 runs live):** [`anuragagarwal203-cisco/clarify-rl`](https://wandb.ai/anuragagarwal203-cisco/clarify-rl)
- **Environment Space (OpenEnv MCP server):** [`agarwalanu3103/clarify-rl`](https://huggingface.co/spaces/agarwalanu3103/clarify-rl)
- **Hackathon write-up (blog):** [`docs/blog.md`](https://github.com/anurag203/clarify-rl/blob/main/docs/blog.md)
## Model summary
| Field | Value |
| --- | --- |
| Base model | [`Qwen/Qwen3-1.7B`](https://huggingface.co/Qwen/Qwen3-1.7B) (NOT the instruct tune) |
| Algorithm | TRL GRPO (Group Relative Policy Optimization) |
| KL anchor (β) | **0.2** (vs Run 2 = 0.0) |
| Learning rate | 5e-7, cosine decay → 1.7e-9 |
| Steps | 300 |
| Wall time | 78.2 min on a single A100 (HF Jobs `a100-large`) |
| Optimizer | adamw_torch_fused |
| Generations / step | 4 (with vLLM sampler) |
| Max completion length | 768 tokens |
| Reward stack | `OutputCorrectnessRubric` (0.6) + `EfficiencyRubric` (0.2) + `FormatCheckRubric` (0.2) over 5 task families |
| Cost | **~$1.80** of HF Jobs credit |
The training and eval are fully reproducible from
[`training/train_grpo.py`](https://github.com/anurag203/clarify-rl/blob/main/training/train_grpo.py)
with `BETA=0.2 LEARNING_RATE=5e-7 NUM_STEPS=300`.
## What this model does
ClarifyRL trains a small LLM to **ask before it acts**. Given a deliberately
under-specified user request (e.g. *"set up a celebration"*), the agent has
6 question budget and must choose between:
- `ask_question(question)` — pull a single missing field from the user simulator
- `propose_plan(plan)` — emit the final structured JSON plan
- `get_task_info()` — re-read the brief
Reward depends on (a) format correctness, (b) field overlap of the final
plan vs the hidden ground-truth profile, and (c) efficiency (fewer
questions for higher-quality plans). 5 task families: `event_planning`,
`medical_intake`, `meeting_scheduling`, `support_triage`,
`coding`. Held-out eval uses 50 scenarios with the same 4-family coverage
as training.
## The headline result — KL anchor turns the model around
Same base, same data, same step count. The only difference is β.
| Eval metric (n=50, held-out) | 1.7B base | Run 2 (β=0) | **Run 4 (β=0.2) ✅** |
| --- | ---: | ---: | ---: |
| avg_score (μ across 4 families) | 0.067 | 0.029 ↓ | **0.056** ✅ |
| completion_rate | 0.18 | 0.06 | **0.14** |
| event_planning μ | 0.138 | **0.000 ❌** | **0.175 ✅✅** |
| event_planning max | 0.522 | 0.000 | 0.510 |
| meeting_scheduling μ | 0.153 | 0.130 | 0.064 ↓ |
| meeting_scheduling max | 0.500 | **0.725** | 0.350 |
Three observations the hackathon write-up leans on:
1. **GRPO without anchor causes capability collapse.** Run 2 (β=0) drove
`event_planning` from 0.138 → **0.000** mean while inflating one peak
in `meeting_scheduling`. The model traded breadth for an
exploit-and-overfit that the held-out eval flags immediately.
2. **GRPO with KL anchor cleanly improves the protected family.**
Run 4 (β=0.2, lr=5e-7) on the same model recovered avg_score to
**0.056** AND **beat the base on `event_planning`** (0.138 → **0.175**).
The anchor literally fixed Run 2's regression *without* extra data.
3. **The cost is peak capability.** Run 4 dropped
`meeting_scheduling` max from 0.725 (Run 2's gem) to 0.350. KL prevents
the kind of extreme specialization Run 2 leaned on. That's the
trade-off, stated honestly.
For the same-base delta plot
([`plots/06_same_base_delta.png`](https://raw.githubusercontent.com/anurag203/clarify-rl/main/plots/06_same_base_delta.png)),
see the full blog at
[`docs/blog.md`](https://github.com/anurag203/clarify-rl/blob/main/docs/blog.md).
## Intended use
- **Research / hackathon** — reproduce the KL-anchor ablation on a small
reasoner.
- **Demo / education** — illustrate that a 1.7B param model can be steered
toward an *ask-first* policy with a tiny RL budget when KL is enforced.
- **Drop-in replacement for `Qwen/Qwen3-1.7B`** in agentic, multi-turn,
tool-using settings where the agent should clarify ambiguous requests
rather than hallucinate fields.
## Out-of-scope use
- General chat assistant. The reward shaping is highly specific to the
ClarifyRL ask-or-guess setting; do not expect calibration or RLHF-style
helpfulness on open-ended prompts.
- Production / safety-critical / medical / legal. The model has **no** RLHF
safety alignment; the `medical_intake` family in the eval is a *task
scaffold*, not a clinical reasoning task.
- Anything outside English.
## How to use it
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
repo = "anurag203/clarify-rl-run4-qwen3-1.7b-beta0.2"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
mdl = AutoModelForCausalLM.from_pretrained(
repo,
torch_dtype=torch.bfloat16,
device_map="auto",
trust_remote_code=True,
)
SYSTEM = (
"You are an agent that must complete a task by asking clarifying "
"questions and then proposing a structured JSON plan. Tools: "
"ask_question(question), propose_plan(plan), get_task_info(). "
"You have a 6-question budget. Output ONLY one tool call per turn."
)
USER = "Set up a celebration."
prompt = tok.apply_chat_template(
[{"role": "system", "content": SYSTEM}, {"role": "user", "content": USER}],
tokenize=False,
add_generation_prompt=True,
chat_template_kwargs={"enable_thinking": False},
)
out = mdl.generate(
**tok(prompt, return_tensors="pt").to(mdl.device),
max_new_tokens=120,
temperature=0.7,
do_sample=True,
pad_token_id=tok.eos_token_id,
)
print(tok.decode(out[0][tok(prompt, return_tensors="pt")["input_ids"].shape[1]:],
skip_special_tokens=True))
# expected first turn: ask_question("What is the date of the celebration?")
```
For the full multi-turn agent loop with tool parsing, see
[`scripts/eval_agent.py`](https://github.com/anurag203/clarify-rl/blob/main/scripts/eval_agent.py)
or the live tab of the
[`anurag203/clarify-rl-demo`](https://huggingface.co/spaces/anurag203/clarify-rl-demo) Space.
## Training data
Held-out eval uses
[`scenarios/eval_held_out.json`](https://github.com/anurag203/clarify-rl/blob/main/scenarios/eval_held_out.json)
(50 scenarios across 4 families). The training data is procedurally generated
inside the OpenEnv MCP environment and is *not* a static dump — the user
simulator answers each `ask_question` with a sampled ground-truth profile,
so the same scenario can yield different conversations across rollouts.
See
[`scenarios/`](https://github.com/anurag203/clarify-rl/tree/main/scenarios)
and
[`docs/05-scenario-design.md`](https://github.com/anurag203/clarify-rl/blob/main/docs/05-scenario-design.md)
for the full scenario taxonomy.
## Training procedure
| Hyperparameter | Value |
| --- | --- |
| Algorithm | GRPO (TRL ≥ 1.0) |
| `beta` (KL coefficient) | **0.2** |
| Learning rate | 5e-7 (cosine to ~1.7e-9) |
| Total steps | 300 |
| Generations / step | 4 |
| Group size | 4 |
| Max completion length | 768 |
| Sampler | vLLM, `temperature=1.0`, `top_p=1.0` |
| Reference model | Frozen `Qwen/Qwen3-1.7B` (loaded once, never updated) |
| Hardware | 1× A100 80GB (HF Jobs `a100-large`) |
| Wall time | 78.2 min |
| Final reward | 0.0050 mean / 0.114 max |
| KL stayed bounded | 0.005 – 0.010 throughout (the anchor did its job) |
| LR cosine-decayed | yes, to 1.7e-9 (verified in W&B) |
The full 300-entry `log_history.json` is committed at
[`outputs/run_artifacts/1.7B-KL/`](https://github.com/anurag203/clarify-rl/tree/main/outputs/run_artifacts/1.7B-KL)
and is mirrored to
[`wandb.ai/anuragagarwal203-cisco/clarify-rl`](https://wandb.ai/anuragagarwal203-cisco/clarify-rl)
under run name `run4-1p7b-kl-anchor`.
## Evaluation
| Family | 1.7B base μ | Run 2 (β=0) μ | **Run 4 (β=0.2) μ** |
| --- | ---: | ---: | ---: |
| event_planning | 0.138 | 0.000 | **0.175** |
| meeting_scheduling | 0.153 | 0.130 | 0.064 |
| medical_intake | 0.000 | 0.000 | 0.000 |
| support_triage | 0.000 | 0.000 | 0.000 |
| **avg_score (μ)** | **0.067** | **0.029** | **0.056** |
| **completion_rate** | 18% | 6% | **14%** |
Eval methodology: 50 held-out scenarios per family, 5 families, scored by
the same rubric stack used at training time. Eval runs are reproducible
from
[`scripts/run_eval.py`](https://github.com/anurag203/clarify-rl/blob/main/scripts/run_eval.py)
with the JSON profile committed at
[`outputs/run_artifacts/1.7B-KL/evals/`](https://github.com/anurag203/clarify-rl/tree/main/outputs/run_artifacts/1.7B-KL).
## Limitations
- 1.7B parameters — not a strong reasoner. Even Run 4's mean (0.056) is
below 4B base's mean (0.145). RL helped, but parameter count still wins.
- Format-pass rate is **0%** across all evaluated runs because the eval
rubric expects strict JSON keys; even Run 4 occasionally proposes plans
that don't quite match the schema. We deliberately leave the format
rubric strict because format failure is a common failure mode of small
RLHF'd models.
- `medical_intake` and `support_triage` remain at 0 across all 1.7B and
0.6B variants — these families need either a stronger base or scenario
redesign. Logged as future work in
[`docs/blog.md` §7b](https://github.com/anurag203/clarify-rl/blob/main/docs/blog.md).
- 4B GRPO (Run 3) was canceled in HF Jobs queue at 48 min — the anchor
finding has not yet been confirmed at 4B scale.
## Citation
If you build on this, please cite the GitHub repository and the W&B
project:
```bibtex
@misc{agarwal2026clarifyrl,
author = {Agarwal, Anurag},
title = {ClarifyRL: Teaching small LLMs to ask before they act,
with KL-anchored GRPO},
year = {2026},
howpublished = {\url{https://github.com/anurag203/clarify-rl}},
note = {Hackathon submission, Apr 26 2026.}
}
```
Built on top of [TRL](https://github.com/huggingface/trl) (GRPO trainer),
[Qwen3](https://huggingface.co/Qwen) (base model), and the [OpenEnv MCP
environment](https://huggingface.co/spaces/agarwalanu3103/clarify-rl).
## License
Apache-2.0 — same as the upstream Qwen3-1.7B base.
Reference in New Issue Copy Permalink