45 lines
3.1 KiB
Markdown
45 lines
3.1 KiB
Markdown
|
=== NPC Reason — Dispatch 3: RLVR / GRPO ===
|
||
|
|
|
||
|
|
Reward: frozen verifier d5d146cf (byte-identical to baseline/SFT), correctness-gated
|
||
|
|
(verifiable&correct=1.0 | correct-only=0.3 | wrong=0.0), hack-guards + stats active.
|
||
|
|
RL data: 5,499 train-split prompts = problems.jsonl MINUS sft_train ids (SFT-rejected;
|
||
|
|
gold known; disjoint from SFT-selection). RL-dev 200 (never eval). decontam vs
|
||
|
|
EVAL.lock = 0 overlap. RL_DATA.lock frozen.
|
||
|
|
GRPO: from SFT (PEFT LoRA r32, ref=base w/ adapter disabled), use_vllm=False (TRL1.6 vs
|
||
|
|
vLLM0.23 mismatch avoided), KL beta=0.08, LR 2e-6, group 8, max_steps 300.
|
||
|
|
best-ckpt by RL-dev verified-and-correct kept.
|
||
|
|
Stability: STABLE — KL 0.0001-0.0004 throughout, no length runaway, no mode collapse, no
|
||
|
|
early-stop trip. (Run hit a transient error at step 260; the best checkpoint is
|
||
|
|
step 200, predates it, unaffected.) This is the regime the 5 prior RL runs never
|
||
|
|
reached: a hard verifier reward trains a small model stably.
|
||
|
|
Dev (hard SFT-rejected tail, 400-tok cap — pessimistic proxy): v&c 0.292 -> 0.323.
|
||
|
|
|
||
|
|
Result (format-prompt, frozen EVAL.lock, greedy, frozen verifier):
|
||
|
|
base SFT NPC-Reason-RL
|
||
|
|
verifiable 0.0% 76.8% 76.2% (RL vs SFT -0.6pp)
|
||
|
|
accuracy 61.6% 65.8% 66.6% (RL vs SFT +0.8pp)
|
||
|
|
verified+corr 0.0% 58.0% 59.6% (RL vs SFT +1.6pp)
|
||
|
|
|
||
|
|
PREREG (final NPC Reason = RL):
|
||
|
|
primary lift verified-and-correct vs base = +59.6pp -> SUPPORTED (bar +15)
|
||
|
|
verifiable-rate = 76.2% -> NOT SUPPORTED (bar 90%)
|
||
|
|
accuracy guard vs base = +5.0pp -> SUPPORTED (no regress)
|
||
|
|
=> 2 of 3 SUPPORTED — same as SFT; RL did NOT move criterion-2 (90% verifiable).
|
||
|
|
|
||
|
|
Shipped: NPC Reason = the RL model (rl/merged) — marginally best on the headline metric
|
||
|
|
(verified-and-correct 59.6 vs SFT 58.0) and accuracy (66.6 vs 65.8); verifiable a
|
||
|
|
hair lower (76.2 vs 76.8). Honest magnitude: ~+1.6pp v&c on n=500 (~8 problems) —
|
||
|
|
a small but consistent-direction win, NOT a decisive one. SFT model retained as the
|
||
|
|
near-equal fallback.
|
||
|
|
|
||
|
|
Honest: RLVR with the frozen verifier reward was STABLE and added a small refinement, but the
|
||
|
|
SFT distillation did the heavy lifting (0 -> 76.8% verifiable). RL could not push
|
||
|
|
verifiable past ~77% under the deliberately conservative (anti-instability) config;
|
||
|
|
the 90% bar is unmet. Per the dispatch, we did NOT crank LR/steps chasing 90% into the
|
||
|
|
instability that killed prior runs — a stable +1.6pp beats an unstable collapse. Not a
|
||
|
|
null (RL > SFT on the headline), but close to one, and reported as such.
|
||
|
|
|
||
|
|
Decisions np-011..np-014 · all 6 prior locks intact · eval untouched until final · RL_RESULT.lock frozen
|
||
|
|
Project arc: base 0% verifiable -> SFT 76.8% (distill) -> RL 76.2%/59.6% v&c (stable refinement).
|
||
|
|
The verifier-grounding transferred; orchestration of the 90% bar is the open frontier.
|