53 lines
2.7 KiB
Markdown
53 lines
2.7 KiB
Markdown
|
# NPC Reason — Pre-registration of lift criteria (FROZEN before any training)
|
|||
|
|
|
|||
|
|
Committed in Dispatch 1, before any SFT / distillation / RL exists. Same discipline as
|
|||
|
|
BigBugAI v2: the success bars are fixed in advance so the eventual before/after delta means
|
|||
|
|
something. These bars are deltas/targets chosen on principle; they do NOT depend on the
|
|||
|
|
measured baseline values (recorded separately in `reports/baseline.md` / `BASELINE.lock`),
|
|||
|
|
so there is no possibility of tuning the bars to the baseline.
|
|||
|
|
|
|||
|
|
The comparator for the primary bar is the base model's **FORMAT-PROMPT** number — the honest
|
|||
|
|
question is "did training add something that prompting the base alone could not?", not
|
|||
|
|
"did training beat a base that was never asked for the format?".
|
|||
|
|
|
|||
|
|
## Metrics (all from the frozen mechanical verifier, VERIFIER.lock)
|
|||
|
|
|
|||
|
|
- **verifiable-rate** — fraction of chains that are mechanically VERIFIABLE
|
|||
|
|
(≥1 load-bearing `<<EXPR=RESULT>>` assertion, all assertions verify, final answer composes).
|
|||
|
|
- **accuracy** — fraction whose final answer equals gold (independent of verifiable).
|
|||
|
|
- **verified-and-correct** — fraction that are BOTH verifiable AND correct. This is the headline.
|
|||
|
|
|
|||
|
|
## Committed criteria
|
|||
|
|
|
|||
|
|
1. **Primary (lift).** SFT NPC Reason must raise **verified-and-correct** by
|
|||
|
|
**≥ +15 percentage points absolute** over the base model's **format-prompt** verified-and-correct rate.
|
|||
|
|
|
|||
|
|
2. **Verifiable-rate target.** NPC Reason must reach **verifiable-rate ≥ 90%**. The project's
|
|||
|
|
purpose is near-total mechanical grounding of load-bearing steps; a specialized model should
|
|||
|
|
emit the checkable format almost always.
|
|||
|
|
|
|||
|
|
3. **Accuracy guard (no regression trade).** NPC Reason **accuracy** must not regress by more
|
|||
|
|
than **5 percentage points** versus the base model's **format-prompt** accuracy. Verifiability
|
|||
|
|
must not be bought by sacrificing correctness — both axes matter.
|
|||
|
|
|
|||
|
|
SFT passes Dispatch 2 only if (1) AND (2) AND (3) all hold.
|
|||
|
|
|
|||
|
|
4. **RL gating (later dispatch).** The RL phase is gated on SFT meeting criteria 1–3. RL's own
|
|||
|
|
criteria will be pre-registered at that time, evaluated against a held-out slice never used
|
|||
|
|
for SFT model selection. The verifier (VERIFIER.lock) is the RL reward and must remain
|
|||
|
|
byte-identical.
|
|||
|
|
|
|||
|
|
## Honesty clause
|
|||
|
|
|
|||
|
|
Report whatever the numbers say. A **null** — training fails to beat the format-prompt
|
|||
|
|
baseline on the primary bar — is a valid, publishable finding, consistent with this being a
|
|||
|
|
real experiment and not a foregone conclusion. The bars above are not moved after results are
|
|||
|
|
seen; if they are missed, that is the result.
|
|||
|
|
|
|||
|
|
## Frozen references
|
|||
|
|
|
|||
|
|
- Verifier: `verifier/VERIFIER.lock`
|
|||
|
|
- Eval set: `eval/EVAL.lock`
|
|||
|
|
- Baseline: `baseline/BASELINE.lock`
|
|||
|
|
- This pre-registration: `reports/PREREG.lock` (sha256 of this file).
|