3.1 KiB
=== NPC Reason — Dispatch 3: RLVR / GRPO ===
Reward: frozen verifier d5d146cf (byte-identical to baseline/SFT), correctness-gated (verifiable&correct=1.0 | correct-only=0.3 | wrong=0.0), hack-guards + stats active. RL data: 5,499 train-split prompts = problems.jsonl MINUS sft_train ids (SFT-rejected; gold known; disjoint from SFT-selection). RL-dev 200 (never eval). decontam vs EVAL.lock = 0 overlap. RL_DATA.lock frozen. GRPO: from SFT (PEFT LoRA r32, ref=base w/ adapter disabled), use_vllm=False (TRL1.6 vs vLLM0.23 mismatch avoided), KL beta=0.08, LR 2e-6, group 8, max_steps 300. best-ckpt by RL-dev verified-and-correct kept. Stability: STABLE — KL 0.0001-0.0004 throughout, no length runaway, no mode collapse, no early-stop trip. (Run hit a transient error at step 260; the best checkpoint is step 200, predates it, unaffected.) This is the regime the 5 prior RL runs never reached: a hard verifier reward trains a small model stably. Dev (hard SFT-rejected tail, 400-tok cap — pessimistic proxy): v&c 0.292 -> 0.323.
Result (format-prompt, frozen EVAL.lock, greedy, frozen verifier): base SFT NPC-Reason-RL verifiable 0.0% 76.8% 76.2% (RL vs SFT -0.6pp) accuracy 61.6% 65.8% 66.6% (RL vs SFT +0.8pp) verified+corr 0.0% 58.0% 59.6% (RL vs SFT +1.6pp)
PREREG (final NPC Reason = RL): primary lift verified-and-correct vs base = +59.6pp -> SUPPORTED (bar +15) verifiable-rate = 76.2% -> NOT SUPPORTED (bar 90%) accuracy guard vs base = +5.0pp -> SUPPORTED (no regress) => 2 of 3 SUPPORTED — same as SFT; RL did NOT move criterion-2 (90% verifiable).
Shipped: NPC Reason = the RL model (rl/merged) — marginally best on the headline metric (verified-and-correct 59.6 vs SFT 58.0) and accuracy (66.6 vs 65.8); verifiable a hair lower (76.2 vs 76.8). Honest magnitude: ~+1.6pp v&c on n=500 (~8 problems) — a small but consistent-direction win, NOT a decisive one. SFT model retained as the near-equal fallback.
Honest: RLVR with the frozen verifier reward was STABLE and added a small refinement, but the SFT distillation did the heavy lifting (0 -> 76.8% verifiable). RL could not push verifiable past ~77% under the deliberately conservative (anti-instability) config; the 90% bar is unmet. Per the dispatch, we did NOT crank LR/steps chasing 90% into the instability that killed prior runs — a stable +1.6pp beats an unstable collapse. Not a null (RL > SFT on the headline), but close to one, and reported as such.
Decisions np-011..np-014 · all 6 prior locks intact · eval untouched until final · RL_RESULT.lock frozen Project arc: base 0% verifiable -> SFT 76.8% (distill) -> RL 76.2%/59.6% v&c (stable refinement). The verifier-grounding transferred; orchestration of the 90% bar is the open frontier.