npc-reason/prereg/PREREG.md

NPC Reason — Pre-registration of lift criteria (FROZEN before any training)

Committed in Dispatch 1, before any SFT / distillation / RL exists. Same discipline as BigBugAI v2: the success bars are fixed in advance so the eventual before/after delta means something. These bars are deltas/targets chosen on principle; they do NOT depend on the measured baseline values (recorded separately in reports/baseline.md / BASELINE.lock), so there is no possibility of tuning the bars to the baseline.

The comparator for the primary bar is the base model's FORMAT-PROMPT number — the honest question is "did training add something that prompting the base alone could not?", not "did training beat a base that was never asked for the format?".

Metrics (all from the frozen mechanical verifier, VERIFIER.lock)

• verifiable-rate — fraction of chains that are mechanically VERIFIABLE (≥1 load-bearing <<EXPR=RESULT>> assertion, all assertions verify, final answer composes).
• accuracy — fraction whose final answer equals gold (independent of verifiable).
• verified-and-correct — fraction that are BOTH verifiable AND correct. This is the headline.

Committed criteria

1. Primary (lift). SFT NPC Reason must raise verified-and-correct by ≥ +15 percentage points absolute over the base model's format-prompt verified-and-correct rate.

2. Verifiable-rate target. NPC Reason must reach verifiable-rate ≥ 90%. The project's purpose is near-total mechanical grounding of load-bearing steps; a specialized model should emit the checkable format almost always.

3. Accuracy guard (no regression trade). NPC Reason accuracy must not regress by more than 5 percentage points versus the base model's format-prompt accuracy. Verifiability must not be bought by sacrificing correctness — both axes matter.

   SFT passes Dispatch 2 only if (1) AND (2) AND (3) all hold.

4. RL gating (later dispatch). The RL phase is gated on SFT meeting criteria 1–3. RL's own criteria will be pre-registered at that time, evaluated against a held-out slice never used for SFT model selection. The verifier (VERIFIER.lock) is the RL reward and must remain byte-identical.

Honesty clause

Report whatever the numbers say. A null — training fails to beat the format-prompt baseline on the primary bar — is a valid, publishable finding, consistent with this being a real experiment and not a foregone conclusion. The bars above are not moved after results are seen; if they are missed, that is the result.

Frozen references

• Verifier: verifier/VERIFIER.lock
• Eval set: eval/EVAL.lock
• Baseline: baseline/BASELINE.lock
• This pre-registration: reports/PREREG.lock (sha256 of this file).