2.1 KiB
NPC Reason - GGUF decision-fidelity (Dispatch 4, Stage 1)
GGUFs of the shipped RL model (rl/merged), built with llama.cpp (CUDA). Fidelity is validated on
DECISIONS, not just structure (the Fin Nano lesson: a structurally-valid quant can still drift the
model's decisions). Each quant ran the SAME 100 frozen-eval problems greedy via llama-cpp-python
(create_chat_completion, the model's embedded chat template), scored by the FROZEN verifier
(d5d146cf), and compared to the bf16 model on those exact 100 problems.
Tolerance: a quant is fidelity-OK if verified-and-correct is within +/-2pp of bf16.
| model | size | verifiable | verified-and-correct | dVC vs bf16 | fidelity |
|---|---|---|---|---|---|
| bf16 | 3.5 GB | 93.0% | 71.0% | ref | ref |
| f16 | 3.6 GB | 95.0% | 71.0% | +0.0 | OK |
| q8_0 | 1.9 GB | 95.0% | 71.0% | +0.0 | OK (recommended) |
| q4_K_M | 1.1 GB | 95.0% | 77.0% | +6.0 | DRIFTS (flagged) |
(The 100-problem subset is GSM8K-heavy, so the absolute v&c is higher than the full n=500 number of 59.6%; fidelity here is the per-quant DELTA, which is the meaningful quantity.)
Recommendation: q8_0
- q8_0 is behaviorally lossless on this product (verified-and-correct delta 0.0 vs bf16). It is half the size of f16 and reproduces the model's decisions. Use it.
- q4_K_M DRIFTS and is flagged, not recommended. Its verified-and-correct moved +6pp (well past the +/-2pp tolerance). The direction happens to be upward on these 100, but that is a decision CHANGE under quant noise, not fidelity. Per the Fin Nano lesson, do not choose a quant on file size alone; q4_K_M's chains are not the bf16 model's chains. Use only if a 1.1 GB footprint is required and the decision drift is acceptable for the use case.
- f16 is provided as the lossless reference.
Validation harness: gguf/fidelity_eval.py (frozen verifier, greedy, n=100). Raw:
gguf/fidelity_result.json.