34 lines
2.1 KiB
Markdown
34 lines
2.1 KiB
Markdown
# NPC Reason - GGUF decision-fidelity (Dispatch 4, Stage 1)
|
|
|
|
GGUFs of the shipped RL model (`rl/merged`), built with llama.cpp (CUDA). Fidelity is validated on
|
|
DECISIONS, not just structure (the Fin Nano lesson: a structurally-valid quant can still drift the
|
|
model's decisions). Each quant ran the SAME 100 frozen-eval problems greedy via llama-cpp-python
|
|
(`create_chat_completion`, the model's embedded chat template), scored by the FROZEN verifier
|
|
(d5d146cf), and compared to the bf16 model on those exact 100 problems.
|
|
|
|
Tolerance: a quant is fidelity-OK if verified-and-correct is within +/-2pp of bf16.
|
|
|
|
| model | size | verifiable | verified-and-correct | dVC vs bf16 | fidelity |
|
|
|---------|-------:|-----------:|---------------------:|------------:|----------|
|
|
| bf16 | 3.5 GB | 93.0% | 71.0% | ref | ref |
|
|
| f16 | 3.6 GB | 95.0% | 71.0% | +0.0 | OK |
|
|
| **q8_0**| 1.9 GB | 95.0% | 71.0% | +0.0 | **OK (recommended)** |
|
|
| q4_K_M | 1.1 GB | 95.0% | 77.0% | +6.0 | DRIFTS (flagged) |
|
|
|
|
(The 100-problem subset is GSM8K-heavy, so the absolute v&c is higher than the full n=500 number
|
|
of 59.6%; fidelity here is the per-quant DELTA, which is the meaningful quantity.)
|
|
|
|
## Recommendation: q8_0
|
|
|
|
- **q8_0 is behaviorally lossless** on this product (verified-and-correct delta 0.0 vs bf16). It is
|
|
half the size of f16 and reproduces the model's decisions. Use it.
|
|
- **q4_K_M DRIFTS and is flagged, not recommended.** Its verified-and-correct moved +6pp (well past
|
|
the +/-2pp tolerance). The direction happens to be upward on these 100, but that is a
|
|
decision CHANGE under quant noise, not fidelity. Per the Fin Nano lesson, do not choose a quant on
|
|
file size alone; q4_K_M's chains are not the bf16 model's chains. Use only if a 1.1 GB footprint
|
|
is required and the decision drift is acceptable for the use case.
|
|
- f16 is provided as the lossless reference.
|
|
|
|
Validation harness: `gguf/fidelity_eval.py` (frozen verifier, greedy, n=100). Raw:
|
|
`gguf/fidelity_result.json`.
|