68 lines
3.0 KiB
Markdown
68 lines
3.0 KiB
Markdown
|
# NPC Reason 1.5B - Usage
|
||
|
|
|
||
|
|
NPC Reason emits math reasoning where every load-bearing step is a checkable `<<EXPR = RESULT>>`
|
||
|
|
assertion. You prompt for the format, then run the included verifier on the output to confirm the
|
||
|
|
chain mechanically. The verifiable-rate is not the model's opinion; it is re-executed by code.
|
||
|
|
|
||
|
|
## 1. The prompt (use this format instruction verbatim)
|
||
|
|
|
||
|
|
```
|
||
|
|
Solve this math problem. For EVERY load-bearing arithmetic step, write the computation as an
|
||
|
|
inline checkable assertion in the exact form <<EXPR = RESULT>>, where EXPR is the arithmetic
|
||
|
|
expression and RESULT is its value (for example <<3*8 = 24>>). If a quantity is reused, you may
|
||
|
|
name it, e.g. let total = <<3*8 = 24>>, and reference it later as <<total + 6 = 30>>. Do not
|
||
|
|
assert any number that drives the answer without wrapping it in <<...>>. End with the final
|
||
|
|
answer as \boxed{ANSWER}, and make sure it equals the result of your last <<...>> step.
|
||
|
|
|
||
|
|
Problem: <your problem here>
|
||
|
|
```
|
||
|
|
|
||
|
|
Apply the model's chat template (single user turn) and decode greedily (temperature 0) for
|
||
|
|
reproducible chains.
|
||
|
|
|
||
|
|
## 2. Run it
|
||
|
|
|
||
|
|
**vLLM (bf16 merged model):**
|
||
|
|
```python
|
||
|
|
from vllm import LLM, SamplingParams
|
||
|
|
from transformers import AutoTokenizer
|
||
|
|
tok = AutoTokenizer.from_pretrained("npc-reason") # merged model dir
|
||
|
|
llm = LLM(model="npc-reason", dtype="bfloat16")
|
||
|
|
prompt = tok.apply_chat_template(
|
||
|
|
[{"role": "user", "content": FORMAT_INSTRUCTION.format(problem=problem)}],
|
||
|
|
tokenize=False, add_generation_prompt=True)
|
||
|
|
out = llm.generate([prompt], SamplingParams(temperature=0.0, max_tokens=1024))
|
||
|
|
chain = out[0].outputs[0].text
|
||
|
|
```
|
||
|
|
|
||
|
|
**llama.cpp / GGUF (recommended quant per gguf_fidelity.md):**
|
||
|
|
```python
|
||
|
|
from llama_cpp import Llama
|
||
|
|
llm = Llama(model_path="npc-reason-q8_0.gguf", n_gpu_layers=99, n_ctx=2048)
|
||
|
|
out = llm.create_chat_completion(
|
||
|
|
messages=[{"role": "user", "content": FORMAT_INSTRUCTION.format(problem=problem)}],
|
||
|
|
temperature=0.0, max_tokens=1024)
|
||
|
|
chain = out["choices"][0]["message"]["content"]
|
||
|
|
```
|
||
|
|
|
||
|
|
## 3. Verify the chain (this is the point)
|
||
|
|
|
||
|
|
```python
|
||
|
|
from verifier.step_verifier import verify_chain # shipped with the model, frozen d5d146cf
|
||
|
|
rec = verify_chain(chain, gold_answer=known_answer) # gold optional
|
||
|
|
print(rec["verifiable"]) # every load-bearing step re-executed AND composes
|
||
|
|
print(rec["correct"]) # final answer == gold (independent axis)
|
||
|
|
print(rec["verified_and_correct"]) # both
|
||
|
|
print(rec["failures"]) # which step broke and why, if any
|
||
|
|
```
|
||
|
|
|
||
|
|
A chain is VERIFIABLE only if every `<<EXPR=RESULT>>` re-executes correctly under SymPy and the
|
||
|
|
final answer composes from the last step. Filler assertions that do not drive the answer do not
|
||
|
|
count. If `verifiable` is False, inspect `failures`; do not trust the chain.
|
||
|
|
|
||
|
|
## 4. Honest expectations
|
||
|
|
|
||
|
|
- ~77% of format-prompt chains verify; ~23% are the unverified tail. Always check.
|
||
|
|
- Math-first (arithmetic and arithmetic-reducible word problems). Not a general chat model.
|
||
|
|
- The SFT model is statistically equivalent to the shipped RL model and is included as a fallback.
|