41 lines
2.8 KiB
Markdown
41 lines
2.8 KiB
Markdown
|
=== NPC Reason — Dispatch 1: Verifier + Frozen Baseline ===
|
||
|
|
|
||
|
|
Verifier: mechanical <<EXPR=RESULT>> checker (SymPy), pure-code, no LLM judgment.
|
||
|
|
VERIFIER.lock d5d146cf… (also the later RL reward signal).
|
||
|
|
tests: correct / arithmetic-error / fluency-trap / non-composing /
|
||
|
|
verifiable-but-wrong / correct-but-unverifiable / variable-binding /
|
||
|
|
unbound-fail-closed / tolerance(exact+1e-6) / extraction — 14/14 pass.
|
||
|
|
Eval: 500 problems = 300 GSM8K(test) + 200 MATH-500, held-out & frozen.
|
||
|
|
EVAL.lock e1573cab… ; meta records sources/splits/seed(20260615). Never training data.
|
||
|
|
Baseline: DeepSeek-R1-Distill-Qwen-1.5B (MIT), untouched, greedy/deterministic (seed 0,
|
||
|
|
max_tokens 12288, ctx 16384). 1000 chains in 556s. BASELINE.lock frozen.
|
||
|
|
|
||
|
|
plain-prompt format-prompt
|
||
|
|
verifiable-rate 0.0% 0.0% <-- base never emits <<...>> (1/500 contain "<<")
|
||
|
|
accuracy 59.8% 61.6% (floor; greedy repetition leaves 83/500 fmt no-answer)
|
||
|
|
verified-and-correct 0.0% 0.0%
|
||
|
|
mean assertions/chain 0.0 0.0
|
||
|
|
|
||
|
|
Reading: The base does NOT produce mechanically-checkable steps, even when explicitly asked
|
||
|
|
(format-prompt verifiable-rate is 0%, not just plain). The project's headline delta
|
||
|
|
is therefore real and large-headroom: any verifiable-rate NPC Reason reaches is lift
|
||
|
|
over ~0. Accuracy is a separate axis (~60%) and is the guard, not the goal.
|
||
|
|
|
||
|
|
Prereg: FROZEN before training (PREREG.lock b5a49437…). Bars committed:
|
||
|
|
1. primary: verified-and-correct +>=15pp over format baseline (i.e. >=15%)
|
||
|
|
2. verifiable-rate >= 90%
|
||
|
|
3. accuracy must not regress > 5pp below format accuracy 61.6% (i.e. >= 56.6%)
|
||
|
|
4. RL gated on SFT passing 1-3; RL criteria pre-registered later.
|
||
|
|
Honesty clause: a null (training fails to beat format baseline) is a valid finding.
|
||
|
|
|
||
|
|
Next: Dispatch 2 = generate DeepSeek-V4 verifiable <<...>> chains, code-filter against
|
||
|
|
VERIFIER.lock, SFT-distill into R1-Distill-1.5B; decontaminate against EVAL.lock.
|
||
|
|
GPU: inference only; A40 ~4.5GB peak. No training performed this dispatch.
|
||
|
|
|
||
|
|
Notes: - Re-ran baseline at 12288 tokens (from 4096) to remove a truncation artifact;
|
||
|
|
4096 run archived at baseline/baseline_results.trunc4096.json. verifiable-rate
|
||
|
|
0% in BOTH runs (robust). Verifier frozen BEFORE both baseline runs (not tuned to them).
|
||
|
|
- R1-Distill greedy repetition (vendor recommends temp 0.6) leaves ~17% of chains
|
||
|
|
without a final answer; counted wrong, so accuracy is a floor. Same decoding will
|
||
|
|
apply post-training -> apples-to-apples.
|