--- license: apache-2.0 base_model: Qwen/Qwen3-1.7B base_model_relation: finetune language: - en library_name: transformers pipeline_tag: text-generation tags: - text-generation - conversational - reinforcement-learning - rlhf - rl - grpo - trl - kl-anchor - agentic - tool-use - clarifying-questions - ask-vs-guess model-index: - name: clarify-rl-run4-qwen3-1.7b-beta0.2 results: - task: type: text-generation name: ClarifyRL multi-turn ask-or-guess dataset: type: held_out_scenarios name: ClarifyRL eval (50 held-out scenarios, n=50 v4) metrics: - type: avg_score value: 0.0560 name: avg_score (μ over 4 families × 50 scenarios) - type: completion_rate value: 0.14 name: completion_rate - type: family_event_planning_mean value: 0.175 name: event_planning μ - type: family_meeting_scheduling_mean value: 0.064 name: meeting_scheduling μ --- # ClarifyRL — Run 4 — Qwen3-1.7B GRPO (β=0.2 KL anchor) > **The hero checkpoint of the ClarifyRL hackathon submission.** This is > Qwen3-1.7B trained with TRL GRPO and an explicit **KL anchor at β=0.2** > against the frozen base, which is the single change that turned a > capability collapse (Run 2, β=0) into a measurable improvement on the > held-out eval *while preserving breadth across families*. - **Live demo (replay + CPU live chat):** [`anurag203/clarify-rl-demo`](https://huggingface.co/spaces/anurag203/clarify-rl-demo) - **Code (training, eval, plots):** [`github.com/anurag203/clarify-rl`](https://github.com/anurag203/clarify-rl) - **W&B dashboard (all 3 runs live):** [`anuragagarwal203-cisco/clarify-rl`](https://wandb.ai/anuragagarwal203-cisco/clarify-rl) - **Environment Space (OpenEnv MCP server):** [`agarwalanu3103/clarify-rl`](https://huggingface.co/spaces/agarwalanu3103/clarify-rl) - **Hackathon write-up (blog):** [`docs/blog.md`](https://github.com/anurag203/clarify-rl/blob/main/docs/blog.md) ## Model summary | Field | Value | | --- | --- | | Base model | [`Qwen/Qwen3-1.7B`](https://huggingface.co/Qwen/Qwen3-1.7B) (NOT the instruct tune) | | Algorithm | TRL GRPO (Group Relative Policy Optimization) | | KL anchor (β) | **0.2** (vs Run 2 = 0.0) | | Learning rate | 5e-7, cosine decay → 1.7e-9 | | Steps | 300 | | Wall time | 78.2 min on a single A100 (HF Jobs `a100-large`) | | Optimizer | adamw_torch_fused | | Generations / step | 4 (with vLLM sampler) | | Max completion length | 768 tokens | | Reward stack | `OutputCorrectnessRubric` (0.6) + `EfficiencyRubric` (0.2) + `FormatCheckRubric` (0.2) over 5 task families | | Cost | **~$1.80** of HF Jobs credit | The training and eval are fully reproducible from [`training/train_grpo.py`](https://github.com/anurag203/clarify-rl/blob/main/training/train_grpo.py) with `BETA=0.2 LEARNING_RATE=5e-7 NUM_STEPS=300`. ## What this model does ClarifyRL trains a small LLM to **ask before it acts**. Given a deliberately under-specified user request (e.g. *"set up a celebration"*), the agent has 6 question budget and must choose between: - `ask_question(question)` — pull a single missing field from the user simulator - `propose_plan(plan)` — emit the final structured JSON plan - `get_task_info()` — re-read the brief Reward depends on (a) format correctness, (b) field overlap of the final plan vs the hidden ground-truth profile, and (c) efficiency (fewer questions for higher-quality plans). 5 task families: `event_planning`, `medical_intake`, `meeting_scheduling`, `support_triage`, `coding`. Held-out eval uses 50 scenarios with the same 4-family coverage as training. ## The headline result — KL anchor turns the model around Same base, same data, same step count. The only difference is β. | Eval metric (n=50, held-out) | 1.7B base | Run 2 (β=0) | **Run 4 (β=0.2) ✅** | | --- | ---: | ---: | ---: | | avg_score (μ across 4 families) | 0.067 | 0.029 ↓ | **0.056** ✅ | | completion_rate | 0.18 | 0.06 | **0.14** | | event_planning μ | 0.138 | **0.000 ❌** | **0.175 ✅✅** | | event_planning max | 0.522 | 0.000 | 0.510 | | meeting_scheduling μ | 0.153 | 0.130 | 0.064 ↓ | | meeting_scheduling max | 0.500 | **0.725** | 0.350 | Three observations the hackathon write-up leans on: 1. **GRPO without anchor causes capability collapse.** Run 2 (β=0) drove `event_planning` from 0.138 → **0.000** mean while inflating one peak in `meeting_scheduling`. The model traded breadth for an exploit-and-overfit that the held-out eval flags immediately. 2. **GRPO with KL anchor cleanly improves the protected family.** Run 4 (β=0.2, lr=5e-7) on the same model recovered avg_score to **0.056** AND **beat the base on `event_planning`** (0.138 → **0.175**). The anchor literally fixed Run 2's regression *without* extra data. 3. **The cost is peak capability.** Run 4 dropped `meeting_scheduling` max from 0.725 (Run 2's gem) to 0.350. KL prevents the kind of extreme specialization Run 2 leaned on. That's the trade-off, stated honestly. For the same-base delta plot ([`plots/06_same_base_delta.png`](https://raw.githubusercontent.com/anurag203/clarify-rl/main/plots/06_same_base_delta.png)), see the full blog at [`docs/blog.md`](https://github.com/anurag203/clarify-rl/blob/main/docs/blog.md). ## Intended use - **Research / hackathon** — reproduce the KL-anchor ablation on a small reasoner. - **Demo / education** — illustrate that a 1.7B param model can be steered toward an *ask-first* policy with a tiny RL budget when KL is enforced. - **Drop-in replacement for `Qwen/Qwen3-1.7B`** in agentic, multi-turn, tool-using settings where the agent should clarify ambiguous requests rather than hallucinate fields. ## Out-of-scope use - General chat assistant. The reward shaping is highly specific to the ClarifyRL ask-or-guess setting; do not expect calibration or RLHF-style helpfulness on open-ended prompts. - Production / safety-critical / medical / legal. The model has **no** RLHF safety alignment; the `medical_intake` family in the eval is a *task scaffold*, not a clinical reasoning task. - Anything outside English. ## How to use it ```python from transformers import AutoModelForCausalLM, AutoTokenizer import torch repo = "anurag203/clarify-rl-run4-qwen3-1.7b-beta0.2" tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True) mdl = AutoModelForCausalLM.from_pretrained( repo, torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True, ) SYSTEM = ( "You are an agent that must complete a task by asking clarifying " "questions and then proposing a structured JSON plan. Tools: " "ask_question(question), propose_plan(plan), get_task_info(). " "You have a 6-question budget. Output ONLY one tool call per turn." ) USER = "Set up a celebration." prompt = tok.apply_chat_template( [{"role": "system", "content": SYSTEM}, {"role": "user", "content": USER}], tokenize=False, add_generation_prompt=True, chat_template_kwargs={"enable_thinking": False}, ) out = mdl.generate( **tok(prompt, return_tensors="pt").to(mdl.device), max_new_tokens=120, temperature=0.7, do_sample=True, pad_token_id=tok.eos_token_id, ) print(tok.decode(out[0][tok(prompt, return_tensors="pt")["input_ids"].shape[1]:], skip_special_tokens=True)) # expected first turn: ask_question("What is the date of the celebration?") ``` For the full multi-turn agent loop with tool parsing, see [`scripts/eval_agent.py`](https://github.com/anurag203/clarify-rl/blob/main/scripts/eval_agent.py) or the live tab of the [`anurag203/clarify-rl-demo`](https://huggingface.co/spaces/anurag203/clarify-rl-demo) Space. ## Training data Held-out eval uses [`scenarios/eval_held_out.json`](https://github.com/anurag203/clarify-rl/blob/main/scenarios/eval_held_out.json) (50 scenarios across 4 families). The training data is procedurally generated inside the OpenEnv MCP environment and is *not* a static dump — the user simulator answers each `ask_question` with a sampled ground-truth profile, so the same scenario can yield different conversations across rollouts. See [`scenarios/`](https://github.com/anurag203/clarify-rl/tree/main/scenarios) and [`docs/05-scenario-design.md`](https://github.com/anurag203/clarify-rl/blob/main/docs/05-scenario-design.md) for the full scenario taxonomy. ## Training procedure | Hyperparameter | Value | | --- | --- | | Algorithm | GRPO (TRL ≥ 1.0) | | `beta` (KL coefficient) | **0.2** | | Learning rate | 5e-7 (cosine to ~1.7e-9) | | Total steps | 300 | | Generations / step | 4 | | Group size | 4 | | Max completion length | 768 | | Sampler | vLLM, `temperature=1.0`, `top_p=1.0` | | Reference model | Frozen `Qwen/Qwen3-1.7B` (loaded once, never updated) | | Hardware | 1× A100 80GB (HF Jobs `a100-large`) | | Wall time | 78.2 min | | Final reward | 0.0050 mean / 0.114 max | | KL stayed bounded | 0.005 – 0.010 throughout (the anchor did its job) | | LR cosine-decayed | yes, to 1.7e-9 (verified in W&B) | The full 300-entry `log_history.json` is committed at [`outputs/run_artifacts/1.7B-KL/`](https://github.com/anurag203/clarify-rl/tree/main/outputs/run_artifacts/1.7B-KL) and is mirrored to [`wandb.ai/anuragagarwal203-cisco/clarify-rl`](https://wandb.ai/anuragagarwal203-cisco/clarify-rl) under run name `run4-1p7b-kl-anchor`. ## Evaluation | Family | 1.7B base μ | Run 2 (β=0) μ | **Run 4 (β=0.2) μ** | | --- | ---: | ---: | ---: | | event_planning | 0.138 | 0.000 | **0.175** | | meeting_scheduling | 0.153 | 0.130 | 0.064 | | medical_intake | 0.000 | 0.000 | 0.000 | | support_triage | 0.000 | 0.000 | 0.000 | | **avg_score (μ)** | **0.067** | **0.029** | **0.056** | | **completion_rate** | 18% | 6% | **14%** | Eval methodology: 50 held-out scenarios per family, 5 families, scored by the same rubric stack used at training time. Eval runs are reproducible from [`scripts/run_eval.py`](https://github.com/anurag203/clarify-rl/blob/main/scripts/run_eval.py) with the JSON profile committed at [`outputs/run_artifacts/1.7B-KL/evals/`](https://github.com/anurag203/clarify-rl/tree/main/outputs/run_artifacts/1.7B-KL). ## Limitations - 1.7B parameters — not a strong reasoner. Even Run 4's mean (0.056) is below 4B base's mean (0.145). RL helped, but parameter count still wins. - Format-pass rate is **0%** across all evaluated runs because the eval rubric expects strict JSON keys; even Run 4 occasionally proposes plans that don't quite match the schema. We deliberately leave the format rubric strict because format failure is a common failure mode of small RLHF'd models. - `medical_intake` and `support_triage` remain at 0 across all 1.7B and 0.6B variants — these families need either a stronger base or scenario redesign. Logged as future work in [`docs/blog.md` §7b](https://github.com/anurag203/clarify-rl/blob/main/docs/blog.md). - 4B GRPO (Run 3) was canceled in HF Jobs queue at 48 min — the anchor finding has not yet been confirmed at 4B scale. ## Citation If you build on this, please cite the GitHub repository and the W&B project: ```bibtex @misc{agarwal2026clarifyrl, author = {Agarwal, Anurag}, title = {ClarifyRL: Teaching small LLMs to ask before they act, with KL-anchored GRPO}, year = {2026}, howpublished = {\url{https://github.com/anurag203/clarify-rl}}, note = {Hackathon submission, Apr 26 2026.} } ``` Built on top of [TRL](https://github.com/huggingface/trl) (GRPO trainer), [Qwen3](https://huggingface.co/Qwen) (base model), and the [OpenEnv MCP environment](https://huggingface.co/spaces/agarwalanu3103/clarify-rl). ## License Apache-2.0 — same as the upstream Qwen3-1.7B base.