53 lines
2.7 KiB
Markdown
53 lines
2.7 KiB
Markdown
# NPC Reason — Pre-registration of lift criteria (FROZEN before any training)
|
||
|
||
Committed in Dispatch 1, before any SFT / distillation / RL exists. Same discipline as
|
||
BigBugAI v2: the success bars are fixed in advance so the eventual before/after delta means
|
||
something. These bars are deltas/targets chosen on principle; they do NOT depend on the
|
||
measured baseline values (recorded separately in `reports/baseline.md` / `BASELINE.lock`),
|
||
so there is no possibility of tuning the bars to the baseline.
|
||
|
||
The comparator for the primary bar is the base model's **FORMAT-PROMPT** number — the honest
|
||
question is "did training add something that prompting the base alone could not?", not
|
||
"did training beat a base that was never asked for the format?".
|
||
|
||
## Metrics (all from the frozen mechanical verifier, VERIFIER.lock)
|
||
|
||
- **verifiable-rate** — fraction of chains that are mechanically VERIFIABLE
|
||
(≥1 load-bearing `<<EXPR=RESULT>>` assertion, all assertions verify, final answer composes).
|
||
- **accuracy** — fraction whose final answer equals gold (independent of verifiable).
|
||
- **verified-and-correct** — fraction that are BOTH verifiable AND correct. This is the headline.
|
||
|
||
## Committed criteria
|
||
|
||
1. **Primary (lift).** SFT NPC Reason must raise **verified-and-correct** by
|
||
**≥ +15 percentage points absolute** over the base model's **format-prompt** verified-and-correct rate.
|
||
|
||
2. **Verifiable-rate target.** NPC Reason must reach **verifiable-rate ≥ 90%**. The project's
|
||
purpose is near-total mechanical grounding of load-bearing steps; a specialized model should
|
||
emit the checkable format almost always.
|
||
|
||
3. **Accuracy guard (no regression trade).** NPC Reason **accuracy** must not regress by more
|
||
than **5 percentage points** versus the base model's **format-prompt** accuracy. Verifiability
|
||
must not be bought by sacrificing correctness — both axes matter.
|
||
|
||
SFT passes Dispatch 2 only if (1) AND (2) AND (3) all hold.
|
||
|
||
4. **RL gating (later dispatch).** The RL phase is gated on SFT meeting criteria 1–3. RL's own
|
||
criteria will be pre-registered at that time, evaluated against a held-out slice never used
|
||
for SFT model selection. The verifier (VERIFIER.lock) is the RL reward and must remain
|
||
byte-identical.
|
||
|
||
## Honesty clause
|
||
|
||
Report whatever the numbers say. A **null** — training fails to beat the format-prompt
|
||
baseline on the primary bar — is a valid, publishable finding, consistent with this being a
|
||
real experiment and not a foregone conclusion. The bars above are not moved after results are
|
||
seen; if they are missed, that is the result.
|
||
|
||
## Frozen references
|
||
|
||
- Verifier: `verifier/VERIFIER.lock`
|
||
- Eval set: `eval/EVAL.lock`
|
||
- Baseline: `baseline/BASELINE.lock`
|
||
- This pre-registration: `reports/PREREG.lock` (sha256 of this file).
|