53 lines
3.0 KiB
Markdown
53 lines
3.0 KiB
Markdown
# NPC Reason — Baseline (untouched base, FROZEN)
|
|
|
|
Base: **deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B** (MIT), untouched, **greedy / deterministic**
|
|
(temperature 0.0, max_tokens 12288, seed 0).
|
|
Eval: **500** problems (GSM8K test + MATH-500), EVAL.lock `e1573cabaf440105…`.
|
|
Verifier: VERIFIER.lock `d5d146cfbb9a69e1…` (pure-code, frozen).
|
|
Generation wall-clock: 556.4s for 1000 chains (both prompts).
|
|
|
|
| | plain-prompt | format-prompt |
|
|
|--------------------------|---------------:|---------------:|
|
|
| verifiable-rate (%) | 0.0 | 0.0 |
|
|
| accuracy (%) | 59.8 | 61.6 |
|
|
| verified-and-correct (%) | 0.0 | 0.0 |
|
|
| mean assertions/chain | 0.0 | 0.0 |
|
|
|
|
## Prompts (verbatim)
|
|
|
|
**plain:**
|
|
```
|
|
Solve this math problem. Show your work step by step, then give the final answer on the last line as \boxed{{ANSWER}}.
|
|
|
|
Problem: {problem}
|
|
```
|
|
|
|
**format-requesting:**
|
|
```
|
|
Solve this math problem. For EVERY load-bearing arithmetic step, write the computation as an inline checkable assertion in the exact form <<EXPR = RESULT>>, where EXPR is the arithmetic expression and RESULT is its value (for example <<3*8 = 24>>). If a quantity is reused, you may name it, e.g. let total = <<3*8 = 24>>, and reference it later as <<total + 6 = 30>>. Do not assert any number that drives the answer without wrapping it in <<...>>. End with the final answer as \boxed{{ANSWER}}, and make sure it equals the result of your last <<...>> step.
|
|
|
|
Problem: {problem}
|
|
```
|
|
|
|
## Reading this honestly
|
|
|
|
- The **format-prompt** column is the comparator for training: it is the ceiling the base can
|
|
reach by prompting alone. The pre-registered primary bar (+15pp verified-and-correct) is over
|
|
THIS column, not over plain.
|
|
- The base's verifiable-rate is whatever the table says — measured, not assumed. If it is already
|
|
high under format-prompt, the project's framing must reflect the real (smaller) delta.
|
|
- R1-Distill is a reasoning model; vendor recommends temp 0.6, but greedy is used here for a reproducible frozen baseline (repetition is possible; reported as-is).
|
|
|
|
This baseline is FROZEN (BASELINE.lock). Every training dispatch compares against exactly these numbers.
|
|
|
|
## Greedy no-answer note (honest)
|
|
|
|
83/500 format-prompt chains (and a similar share of plain) produced NO extractable final
|
|
answer within the 12,288-token budget — these are R1-Distill greedy **repetition loops**
|
|
(the vendor recommends temperature 0.6 for exactly this reason). We keep GREEDY for a
|
|
reproducible frozen baseline and count these as wrong; accuracy here is therefore a FLOOR.
|
|
Raising the budget 4096->12288 only moved no-answer 95->83, confirming the cause is
|
|
degeneration, not truncation. The SAME greedy decoding will be applied to NPC Reason after
|
|
training, so the before/after comparison is apples-to-apples (and training is expected to
|
|
reduce the looping). A prior 4096-token run is archived at baseline/baseline_results.trunc4096.json.
|