ramankrishna10/npc-reason
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: ramankrishna10/npc-reason
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-07-18 02:51:09 +08:00
commit 626c71c533
33 changed files with 1887 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

52
reports/baseline.md Normal file
View File

@@ -0,0 +1,52 @@
# NPC Reason — Baseline (untouched base, FROZEN)
Base: **deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B** (MIT), untouched, **greedy / deterministic**
(temperature 0.0, max_tokens 12288, seed 0).
Eval: **500** problems (GSM8K test + MATH-500), EVAL.lock `e1573cabaf440105…`.
Verifier: VERIFIER.lock `d5d146cfbb9a69e1…` (pure-code, frozen).
Generation wall-clock: 556.4s for 1000 chains (both prompts).
| | plain-prompt | format-prompt |
|--------------------------|---------------:|---------------:|
| verifiable-rate (%) | 0.0 | 0.0 |
| accuracy (%) | 59.8 | 61.6 |
| verified-and-correct (%) | 0.0 | 0.0 |
| mean assertions/chain | 0.0 | 0.0 |
## Prompts (verbatim)
**plain:**
```
Solve this math problem. Show your work step by step, then give the final answer on the last line as \boxed{{ANSWER}}.
Problem: {problem}
```
**format-requesting:**
```
Solve this math problem. For EVERY load-bearing arithmetic step, write the computation as an inline checkable assertion in the exact form <<EXPR = RESULT>>, where EXPR is the arithmetic expression and RESULT is its value (for example <<3*8 = 24>>). If a quantity is reused, you may name it, e.g. let total = <<3*8 = 24>>, and reference it later as <<total + 6 = 30>>. Do not assert any number that drives the answer without wrapping it in <<...>>. End with the final answer as \boxed{{ANSWER}}, and make sure it equals the result of your last <<...>> step.
Problem: {problem}
```
## Reading this honestly
- The **format-prompt** column is the comparator for training: it is the ceiling the base can
reach by prompting alone. The pre-registered primary bar (+15pp verified-and-correct) is over
THIS column, not over plain.
- The base's verifiable-rate is whatever the table says — measured, not assumed. If it is already
high under format-prompt, the project's framing must reflect the real (smaller) delta.
- R1-Distill is a reasoning model; vendor recommends temp 0.6, but greedy is used here for a reproducible frozen baseline (repetition is possible; reported as-is).
This baseline is FROZEN (BASELINE.lock). Every training dispatch compares against exactly these numbers.
## Greedy no-answer note (honest)
83/500 format-prompt chains (and a similar share of plain) produced NO extractable final
answer within the 12,288-token budget — these are R1-Distill greedy **repetition loops**
(the vendor recommends temperature 0.6 for exactly this reason). We keep GREEDY for a
reproducible frozen baseline and count these as wrong; accuracy here is therefore a FLOOR.
Raising the budget 4096->12288 only moved no-answer 95->83, confirming the cause is
degeneration, not truncation. The SAME greedy decoding will be applied to NPC Reason after
training, so the before/after comparison is apples-to-apples (and training is expected to
reduce the looping). A prior 4096-token run is archived at baseline/baseline_results.trunc4096.json.

40
reports/dispatch1_complete.md Normal file
View File

@@ -0,0 +1,40 @@
=== NPC Reason — Dispatch 1: Verifier + Frozen Baseline ===
Verifier: mechanical <<EXPR=RESULT>> checker (SymPy), pure-code, no LLM judgment.
VERIFIER.lock d5d146cf… (also the later RL reward signal).
tests: correct / arithmetic-error / fluency-trap / non-composing /
verifiable-but-wrong / correct-but-unverifiable / variable-binding /
unbound-fail-closed / tolerance(exact+1e-6) / extraction — 14/14 pass.
Eval: 500 problems = 300 GSM8K(test) + 200 MATH-500, held-out & frozen.
EVAL.lock e1573cab… ; meta records sources/splits/seed(20260615). Never training data.
Baseline: DeepSeek-R1-Distill-Qwen-1.5B (MIT), untouched, greedy/deterministic (seed 0,
max_tokens 12288, ctx 16384). 1000 chains in 556s. BASELINE.lock frozen.
plain-prompt format-prompt
verifiable-rate 0.0% 0.0% <-- base never emits <<...>> (1/500 contain "<<")
accuracy 59.8% 61.6% (floor; greedy repetition leaves 83/500 fmt no-answer)
verified-and-correct 0.0% 0.0%
mean assertions/chain 0.0 0.0
Reading: The base does NOT produce mechanically-checkable steps, even when explicitly asked
(format-prompt verifiable-rate is 0%, not just plain). The project's headline delta
is therefore real and large-headroom: any verifiable-rate NPC Reason reaches is lift
over ~0. Accuracy is a separate axis (~60%) and is the guard, not the goal.
Prereg: FROZEN before training (PREREG.lock b5a49437…). Bars committed:
1. primary: verified-and-correct +>=15pp over format baseline (i.e. >=15%)
2. verifiable-rate >= 90%
3. accuracy must not regress > 5pp below format accuracy 61.6% (i.e. >= 56.6%)
4. RL gated on SFT passing 1-3; RL criteria pre-registered later.
Honesty clause: a null (training fails to beat format baseline) is a valid finding.
Next: Dispatch 2 = generate DeepSeek-V4 verifiable <<...>> chains, code-filter against
VERIFIER.lock, SFT-distill into R1-Distill-1.5B; decontaminate against EVAL.lock.
GPU: inference only; A40 ~4.5GB peak. No training performed this dispatch.
Notes: - Re-ran baseline at 12288 tokens (from 4096) to remove a truncation artifact;
4096 run archived at baseline/baseline_results.trunc4096.json. verifiable-rate
0% in BOTH runs (robust). Verifier frozen BEFORE both baseline runs (not tuned to them).
- R1-Distill greedy repetition (vendor recommends temp 0.6) leaves ~17% of chains
without a final answer; counted wrong, so accuracy is a floor. Same decoding will
apply post-training -> apples-to-apples.

36
reports/dispatch2_complete.md Normal file
View File

@@ -0,0 +1,36 @@
=== NPC Reason — Dispatch 2: V4 Corpus + SFT Distill ===
Corpus: 13,245 V4 (deepseek-v4-flash) chains from TRAIN splits (GSM8K-train + MATH-train 7 subj)
-> mechanically filtered against FROZEN verifier (verifiable AND correct)
-> 7,546 kept (keep rate 56.97%). TRAINSET.lock frozen.
reject breakdown: no_assertions 3894 (29%, mostly non-arithmetic MATH) ·
bad_arithmetic 1211 (9%) · non_composing 387 · wrong_answer 206 · dup 1
decontam vs EVAL.lock: 0 overlap (train/test disjoint by construction)
Finding: the strong teacher emits ungrounded/wrong chains ~43% of the time on this
distribution; all rejected before training (the strict-basis move).
SFT: R1-Distill-1.5B + LoRA r64 (7 modules, 73.9M = 3.99% trainable), batch1 x accum16,
3 epochs, seq 1024 constant-pad (0.03% truncated), sdpa + gradient checkpointing.
train loss 0.538 -> 0.075, eval 0.311, runtime 2h31m. Merged model saved (3.5GB).
Result: (format-prompt, vs frozen baseline, greedy, frozen verifier)
verifiable 76.8% (base 0.0) · accuracy 65.8% (base 61.6) · verified+correct 58.0% (base 0.0)
mean assertions/chain 3.44 (base 0.0)
PREREG: lift +58.0pp verified-and-correct -> SUPPORTED (bar +15pp)
verifiable-rate 76.8% -> NOT SUPPORTED (bar 90%)
accuracy delta +4.2pp (improved) -> SUPPORTED (bar: no regress >5pp)
=> 2 of 3 SUPPORTED; all-three gate not met.
RL gate: WARRANTED via the high-signal condition (76.8% verifiable / 58% v&c gives the
verifier-reward RL dense signal); all-three bar NOT cleared (verifiable < 90 is the
explicit RL target). Not a null — a large real lift missing one of three bars.
Honest: Thesis validated — distilling ONLY verifier-confirmed chains took the model from 0%
mechanically-checkable reasoning to 76.8% verifiable / 58% verified-and-correct, with
accuracy rising not falling. The grounding transferred; the remaining ~23% unverifiable
tail is the RL phase's job.
Decisions np-007..np-010 · all frozen locks intact · eval decontaminated · SFT_RESULT.lock frozen
Next: Dispatch 3 = RLVR/GRPO with the frozen verifier as reward, target verifiable-rate >=90%,
RL criteria pre-registered against a held-out slice never used for SFT selection.

44
reports/dispatch3_complete.md Normal file
View File

@@ -0,0 +1,44 @@
=== NPC Reason — Dispatch 3: RLVR / GRPO ===
Reward: frozen verifier d5d146cf (byte-identical to baseline/SFT), correctness-gated
(verifiable&correct=1.0 | correct-only=0.3 | wrong=0.0), hack-guards + stats active.
RL data: 5,499 train-split prompts = problems.jsonl MINUS sft_train ids (SFT-rejected;
gold known; disjoint from SFT-selection). RL-dev 200 (never eval). decontam vs
EVAL.lock = 0 overlap. RL_DATA.lock frozen.
GRPO: from SFT (PEFT LoRA r32, ref=base w/ adapter disabled), use_vllm=False (TRL1.6 vs
vLLM0.23 mismatch avoided), KL beta=0.08, LR 2e-6, group 8, max_steps 300.
best-ckpt by RL-dev verified-and-correct kept.
Stability: STABLE — KL 0.0001-0.0004 throughout, no length runaway, no mode collapse, no
early-stop trip. (Run hit a transient error at step 260; the best checkpoint is
step 200, predates it, unaffected.) This is the regime the 5 prior RL runs never
reached: a hard verifier reward trains a small model stably.
Dev (hard SFT-rejected tail, 400-tok cap — pessimistic proxy): v&c 0.292 -> 0.323.
Result (format-prompt, frozen EVAL.lock, greedy, frozen verifier):
base SFT NPC-Reason-RL
verifiable 0.0% 76.8% 76.2% (RL vs SFT -0.6pp)
accuracy 61.6% 65.8% 66.6% (RL vs SFT +0.8pp)
verified+corr 0.0% 58.0% 59.6% (RL vs SFT +1.6pp)
PREREG (final NPC Reason = RL):
primary lift verified-and-correct vs base = +59.6pp -> SUPPORTED (bar +15)
verifiable-rate = 76.2% -> NOT SUPPORTED (bar 90%)
accuracy guard vs base = +5.0pp -> SUPPORTED (no regress)
=> 2 of 3 SUPPORTED — same as SFT; RL did NOT move criterion-2 (90% verifiable).
Shipped: NPC Reason = the RL model (rl/merged) — marginally best on the headline metric
(verified-and-correct 59.6 vs SFT 58.0) and accuracy (66.6 vs 65.8); verifiable a
hair lower (76.2 vs 76.8). Honest magnitude: ~+1.6pp v&c on n=500 (~8 problems) —
a small but consistent-direction win, NOT a decisive one. SFT model retained as the
near-equal fallback.
Honest: RLVR with the frozen verifier reward was STABLE and added a small refinement, but the
SFT distillation did the heavy lifting (0 -> 76.8% verifiable). RL could not push
verifiable past ~77% under the deliberately conservative (anti-instability) config;
the 90% bar is unmet. Per the dispatch, we did NOT crank LR/steps chasing 90% into the
instability that killed prior runs — a stable +1.6pp beats an unstable collapse. Not a
null (RL > SFT on the headline), but close to one, and reported as such.
Decisions np-011..np-014 · all 6 prior locks intact · eval untouched until final · RL_RESULT.lock frozen
Project arc: base 0% verifiable -> SFT 76.8% (distill) -> RL 76.2%/59.6% v&c (stable refinement).
The verifier-grounding transferred; orchestration of the 90% bar is the open frontier.

48
reports/dispatch4_complete.md Normal file
View File

@@ -0,0 +1,48 @@
=== NPC Reason - Dispatch 4: Packaging + Card ===
GGUF: f16 / q8_0 / q4_K_M of the shipped RL model (rl/merged), built with llama.cpp (CUDA).
Decision-fidelity vs bf16 on 100 frozen-eval problems (greedy, frozen verifier d5d146cf):
f16 dVC +0.0 OK
q8_0 dVC +0.0 OK <- RECOMMENDED (behaviorally lossless, 1.9 GB)
q4_K_M dVC +6.0 DRIFTS (exceeds +/-2pp) -> flagged, NOT recommended
The Fin Nano lesson reproduced: q4 is structurally valid but changes decisions under quant
noise. Do not pick a quant on file size alone. (reports/gguf_fidelity.md)
Card: MODEL_CARD.md - full base/SFT/RL arc (verifiable 0 -> 76.8 -> 76.2; v&c 0 -> 58.0 -> 59.6;
accuracy 61.6 -> 65.8 -> 66.6), distillation-did-the-heavy-lifting, RLVR-stable-but-small
(~tied with SFT), 90% verifiable bar NOT met (stated as the open limitation), verifier
d5d146cf documented as BOTH the metric AND the reward, MIT/V4 lineage, attribution
Rama Krishna Bachu / Bottensor / ORCID 0009-0000-1298-0681. No overclaiming, no em dashes.
USAGE.md - format prompt + vLLM/GGUF run + how to run the verifier on outputs.
Release tree (release/, 22 curated text files; model + GGUF binaries referenced separately):
MODEL_CARD.md, USAGE.md
verifier/ (step_verifier.py + VERIFIER.lock) <- the product: users verify chains
eval/ (run_eval.py, eval_set.jsonl, _eval_meta.json, EVAL.lock)
tests/ (test_verifier.py)
prereg/ (PREREG.md + PREREG.lock)
locks/ (BASELINE / TRAINSET / SFT_RESULT / RL_RESULT .lock)
reports/ (baseline, sft_result, dispatch1/2/3_complete, gguf_fidelity, fidelity_result.json)
MODEL BINARIES (push-time): rl/merged/ (bf16, shipped) + gguf/{f16,q8_0,q4_K_M}.
EXCLUDED (de-link / methodology / secrets-touching): decisions.md, data/generate_v4.py
(DeepSeek API code), all *.sh run/build scripts, raw logs, reward_stats, training scripts.
Credential scan: CLEAN (HARD-HALT gate PASSED). Adversarial multi-angle audit (5 independent
auditors + reconciling synthesizer, all CLEAN, independently re-verified on the host):
- strict secret regex (sk-/hf_/ghp_/gho_/AKIA/xox*/AIza/BEGIN PRIVATE KEY): zero matches.
- all 5 known chat-leaked key prefixes (DeepSeek / HF x2 / CoinGecko / OpenAI-style): absent.
- "DEEPSEEK_API_KEY" string: absent from release/ (only in excluded generate_v4.py + decisions.md).
- release .py files: local model-load + SymPy only, no os.environ/network/auth.
- false-positives dismissed: SHA-256 hashes, env-var NAMES, tokenizer vocab tokens, a GSM8K
"secret seasoning" word problem.
Push: NOT done (prep only, per dispatch). Preconditions before a push dispatch runs:
1. Operator supplies HF org/repo name and confirms PRIVATE.
2. Operator ROTATES the DeepSeek key (still outstanding; it passed through chat).
3. At push time, re-confirm a FULL (not sampled) credential scan against the EXACT push
manifest IF model binaries (gguf/safetensors) are bundled (they live outside release/
and were sampled-clean: safetensors header no-secret, gguf strings clean).
4. Apply the same de-link discipline used for the BigBugAI HF push (no lineage docs).
Decisions np-015..np-018 logged. 7 frozen locks intact (no training/eval-logic changes this dispatch).
Project complete: base 0% -> SFT 76.8% -> RL 76.2%/59.6% v&c, verifier-grounded, honestly reported.

30
reports/fidelity_result.json Normal file
View File

@@ -0,0 +1,30 @@
{
"bf16_reference_n100": {
"n": 100,
"verifiable": 93.0,
"vc": 71.0
},
"tolerance_pp": 2.0,
"quants": {
"f16": {
"verifiable": 95.0,
"vc": 71.0,
"vc_delta_vs_bf16": 0.0,
"fidelity_ok": true
},
"q8_0": {
"verifiable": 95.0,
"vc": 71.0,
"vc_delta_vs_bf16": 0.0,
"fidelity_ok": true
},
"q4_K_M": {
"verifiable": 95.0,
"vc": 77.0,
"vc_delta_vs_bf16": 6.0,
"fidelity_ok": false
}
},
"recommended": "q8_0",
"q4_drifts": true
}

33
reports/gguf_fidelity.md Normal file
View File

@@ -0,0 +1,33 @@
# NPC Reason - GGUF decision-fidelity (Dispatch 4, Stage 1)
GGUFs of the shipped RL model (`rl/merged`), built with llama.cpp (CUDA). Fidelity is validated on
DECISIONS, not just structure (the Fin Nano lesson: a structurally-valid quant can still drift the
model's decisions). Each quant ran the SAME 100 frozen-eval problems greedy via llama-cpp-python
(`create_chat_completion`, the model's embedded chat template), scored by the FROZEN verifier
(d5d146cf), and compared to the bf16 model on those exact 100 problems.
Tolerance: a quant is fidelity-OK if verified-and-correct is within +/-2pp of bf16.
| model | size | verifiable | verified-and-correct | dVC vs bf16 | fidelity |
|---------|-------:|-----------:|---------------------:|------------:|----------|
| bf16 | 3.5 GB | 93.0% | 71.0% | ref | ref |
| f16 | 3.6 GB | 95.0% | 71.0% | +0.0 | OK |
| **q8_0**| 1.9 GB | 95.0% | 71.0% | +0.0 | **OK (recommended)** |
| q4_K_M | 1.1 GB | 95.0% | 77.0% | +6.0 | DRIFTS (flagged) |
(The 100-problem subset is GSM8K-heavy, so the absolute v&c is higher than the full n=500 number
of 59.6%; fidelity here is the per-quant DELTA, which is the meaningful quantity.)
## Recommendation: q8_0
- **q8_0 is behaviorally lossless** on this product (verified-and-correct delta 0.0 vs bf16). It is
half the size of f16 and reproduces the model's decisions. Use it.
- **q4_K_M DRIFTS and is flagged, not recommended.** Its verified-and-correct moved +6pp (well past
the +/-2pp tolerance). The direction happens to be upward on these 100, but that is a
decision CHANGE under quant noise, not fidelity. Per the Fin Nano lesson, do not choose a quant on
file size alone; q4_K_M's chains are not the bf16 model's chains. Use only if a 1.1 GB footprint
is required and the decision drift is acceptable for the use case.
- f16 is provided as the lossless reference.
Validation harness: `gguf/fidelity_eval.py` (frozen verifier, greedy, n=100). Raw:
`gguf/fidelity_result.json`.

53
reports/sft_result.md Normal file
View File

@@ -0,0 +1,53 @@
# NPC Reason — SFT result vs frozen baseline + PREREG (Dispatch 2, Stage 4)
NPC-Reason-SFT = R1-Distill-1.5B + LoRA (r64), distilled on **7,546 V4 chains that the FROZEN
verifier confirmed verifiable AND correct**. Evaluated on the frozen EVAL.lock (500 held-out
GSM8K+MATH), same greedy decoding as the baseline (apples-to-apples), scored by the frozen
verifier (VERIFIER.lock d5d146cf).
## Result (format-prompt = the committed comparator)
| | base (format) | NPC-Reason-SFT | delta |
|------------------------|--------------:|---------------:|--------:|
| verifiable-rate | 0.0% | **76.8%** | +76.8pp |
| accuracy | 61.6% | **65.8%** | +4.2pp |
| verified-and-correct | 0.0% | **58.0%** | +58.0pp |
| mean assertions/chain | 0.0 | 3.44 | — |
Plain-prompt (native, no format instruction): verifiable 0.0%, accuracy 64.8%. The model emits
the `<<EXPR=RESULT>>` format under the format prompt it was trained on, not unprompted — expected,
and the format prompt is the committed comparator.
## PREREG verdict (committed bars, frozen b5a49437 — graded honestly)
| Criterion | Bar | Result | Verdict |
|---|---|---|---|
| 1. Primary lift (verified-and-correct over format baseline) | ≥ +15pp | **+58.0pp** | **SUPPORTED** |
| 2. Verifiable-rate | ≥ 90% | 76.8% | **NOT SUPPORTED** |
| 3. Accuracy guard (no regression > 5pp) | ≥ −5pp | **+4.2pp** (improved) | **SUPPORTED** |
**2 of 3 SUPPORTED.** The all-three gate is not met (criterion 2 missed).
## Honest read
- **The central thesis is validated, strongly.** Distilling *only mechanically-verified* teacher
chains moved a model that produced **zero** checkable reasoning (0% verifiable, even when asked)
to **76.8% verifiable / 58% verified-and-correct** — a +58pp jump on the headline metric, far
past the +15pp bar. The grounding transferred.
- **Accuracy did not regress — it rose** (+4.2pp). Verifiability was not bought by sacrificing
correctness; the strict-basis corpus improved both. (Some of the rise is reduced greedy
repetition vs the base; attributed, not overclaimed.)
- **The one missed bar is verifiable-rate (76.8% vs 90%).** ~23% of format-prompt chains still
fail full verification — a step that doesn't verify, doesn't compose, or the model reverts to
prose on harder MATH items. This is exactly the gap RL-with-verifier-reward is designed to close.
## RL gate
The committed gate is: RL proceeds if SFT clears its bar **OR** produces a high verifiable-rate
so RL has signal to sharpen. All-three bar: **NOT cleared** (verifiable 76.8 < 90). High-signal
condition: **MET** — 76.8% verifiable / 58% verified-and-correct gives RL (verifier = reward, the
same frozen code) dense signal to push toward ≥90%. So the RL phase is **warranted**, entered with
eyes open that criterion 2 is the explicit target. A null at SFT would have been valid; this is not
a null — it is a large, real lift that misses one of three bars.
Frozen: `sft/SFT_RESULT.lock`. Raw per-item outputs: `eval/eval_raw.jsonl`.