初始化项目,由ModelHub XC社区提供模型
Model: ramankrishna10/npc-reason Source: Original Platform
This commit is contained in:
5
prereg/PREREG.lock
Normal file
5
prereg/PREREG.lock
Normal file
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"files": {
|
||||
"reports/PREREG.md": "b5a49437c93e733d49d32b9078f1964bdcef29d41c1148d40f6c54d53493959e"
|
||||
}
|
||||
}
|
||||
52
prereg/PREREG.md
Normal file
52
prereg/PREREG.md
Normal file
@@ -0,0 +1,52 @@
|
||||
# NPC Reason — Pre-registration of lift criteria (FROZEN before any training)
|
||||
|
||||
Committed in Dispatch 1, before any SFT / distillation / RL exists. Same discipline as
|
||||
BigBugAI v2: the success bars are fixed in advance so the eventual before/after delta means
|
||||
something. These bars are deltas/targets chosen on principle; they do NOT depend on the
|
||||
measured baseline values (recorded separately in `reports/baseline.md` / `BASELINE.lock`),
|
||||
so there is no possibility of tuning the bars to the baseline.
|
||||
|
||||
The comparator for the primary bar is the base model's **FORMAT-PROMPT** number — the honest
|
||||
question is "did training add something that prompting the base alone could not?", not
|
||||
"did training beat a base that was never asked for the format?".
|
||||
|
||||
## Metrics (all from the frozen mechanical verifier, VERIFIER.lock)
|
||||
|
||||
- **verifiable-rate** — fraction of chains that are mechanically VERIFIABLE
|
||||
(≥1 load-bearing `<<EXPR=RESULT>>` assertion, all assertions verify, final answer composes).
|
||||
- **accuracy** — fraction whose final answer equals gold (independent of verifiable).
|
||||
- **verified-and-correct** — fraction that are BOTH verifiable AND correct. This is the headline.
|
||||
|
||||
## Committed criteria
|
||||
|
||||
1. **Primary (lift).** SFT NPC Reason must raise **verified-and-correct** by
|
||||
**≥ +15 percentage points absolute** over the base model's **format-prompt** verified-and-correct rate.
|
||||
|
||||
2. **Verifiable-rate target.** NPC Reason must reach **verifiable-rate ≥ 90%**. The project's
|
||||
purpose is near-total mechanical grounding of load-bearing steps; a specialized model should
|
||||
emit the checkable format almost always.
|
||||
|
||||
3. **Accuracy guard (no regression trade).** NPC Reason **accuracy** must not regress by more
|
||||
than **5 percentage points** versus the base model's **format-prompt** accuracy. Verifiability
|
||||
must not be bought by sacrificing correctness — both axes matter.
|
||||
|
||||
SFT passes Dispatch 2 only if (1) AND (2) AND (3) all hold.
|
||||
|
||||
4. **RL gating (later dispatch).** The RL phase is gated on SFT meeting criteria 1–3. RL's own
|
||||
criteria will be pre-registered at that time, evaluated against a held-out slice never used
|
||||
for SFT model selection. The verifier (VERIFIER.lock) is the RL reward and must remain
|
||||
byte-identical.
|
||||
|
||||
## Honesty clause
|
||||
|
||||
Report whatever the numbers say. A **null** — training fails to beat the format-prompt
|
||||
baseline on the primary bar — is a valid, publishable finding, consistent with this being a
|
||||
real experiment and not a foregone conclusion. The bars above are not moved after results are
|
||||
seen; if they are missed, that is the result.
|
||||
|
||||
## Frozen references
|
||||
|
||||
- Verifier: `verifier/VERIFIER.lock`
|
||||
- Eval set: `eval/EVAL.lock`
|
||||
- Baseline: `baseline/BASELINE.lock`
|
||||
- This pre-registration: `reports/PREREG.lock` (sha256 of this file).
|
||||
Reference in New Issue
Block a user