9.6 KiB
Qwen3-0.6B Quantization Comparison Summary
F16 Baseline Reference
| Metric | Value |
|---|---|
| F16 Perplexity | 21.8916 |
| File Size | 1.40 GiB (16.00 BPW) |
All precision loss percentages below are calculated relative to this F16 baseline.
Q4_K_HIFI (INT8 Residuals + Per-Block Scale)
Pros:
- 🏆 Best quality without imatrix with perplexity of 23.66 (0.1% better than Q4_K_M, 3.6% better than Q4_K_S)
- 📊 +8.1% PPL vs F16 — moderate precision loss from quantization
- 🎯 Uses intelligent outlier preservation with INT8 residuals on critical tensors
- 🔬 17 tensors use Q5_K_HIFI_RES8 format for maximum precision on sensitive weights
Cons:
- 💾 Largest file size at 487.39 MiB (+6.9% vs Q4_K_M)
- 🐢 Slower than both Q4_K variants (1.7% slower than Q4_K_M, 3.0% slower than Q4_K_S)
Best for: Production deployments where output quality matters, tasks requiring accuracy (reasoning, coding, complex instructions), especially on smaller models where quantization error has larger impact.
Performance Comparison (Q4_K_HIFI vs the others)
Q4_K_M
| Metric | Q4_K_HIFI | Q4_K_M | Difference |
|---|---|---|---|
| Speed (TPS) | 614.32 | 624.66 | -10.34 (1.7% slower) |
| Perplexity | 23.6556 | 23.6856 | -0.03 (0.1% better) |
| PPL vs F16 | +8.1% | +8.2% | 0.1% less precision loss |
| File Size | 487.39 MiB | 456.11 MiB | +31.28 MiB (6.9% larger) |
| Bits Per Weight | 5.44 | 5.09 | +0.35 (6.9% more) |
Pros:
- ⚖️ Traditional "balanced" approach between speed and quality
- 📚 Well-documented, standard quantization method
- 💾 Smaller file size than Q4_K_HIFI
- ⚡ 1.7% faster inference than Q4_K_HIFI
- 📊 +8.2% PPL vs F16 — nearly identical precision loss to Q4_K_HIFI
Cons:
- ❌ Slightly lower quality (essentially equal - only 0.1% higher perplexity than Q4_K_HIFI)
Best for: When storage is constrained but you still need reasonable quality. Summary: Without imatrix, Q4_K_HIFI and Q4_K_M are essentially identical in quality (0.1% difference). Q4_K_M is smaller (6.9% less) and faster (1.7%), making it the better choice without imatrix.
Q4_K_S
| Metric | Q4_K_HIFI | Q4_K_S | Difference |
|---|---|---|---|
| Speed (TPS) | 614.32 | 632.79 | -18.47 (3.0% slower) |
| Perplexity | 23.6556 | 24.5475 | -0.89 (3.6% better) |
| PPL vs F16 | +8.1% | +12.1% | 4.0% less precision loss |
| File Size | 487.39 MiB | 443.30 MiB | +44.09 MiB (9.9% larger) |
| Bits Per Weight | 5.44 | 4.95 | +0.49 (9.9% more) |
Pros:
- ⚡ Fastest inference at 632.79 TPS (3.0% faster than Q4_K_HIFI)
- 💾 Smallest file size at 443.30 MiB
- ✅ Best choice when speed and storage are critical
Cons:
- ❌ Significantly worse quality with perplexity of 24.55 (3.6% higher than Q4_K_HIFI)
- 📊 +12.1% PPL vs F16 — highest precision loss of all Q4_K variants
- ⚠️ Uses minimal q6_K enhancement — impacts quality on sensitive weights
Best for: Extreme resource constraints, quick prototyping, or bulk processing where quality is less important. Summary: Q4_K_HIFI trades a 3.0% speed reduction and 9.9% larger file size for a 3.6% improvement in quality.
Recommendation Matrix (Without imatrix)
| Priority | Recommended Model | Rationale |
|---|---|---|
| Quality First | Q4_K_HIFI | 3.6% better perplexity than Q4_K_S, essentially equal to Q4_K_M |
| Speed First | Q4_K_S | 3.0% faster, acceptable if quality degradation is tolerable |
| Best Balance | Q4_K_M | Equal quality to Q4_K_HIFI, smaller (6.9%) and faster (1.7%) |
| Smallest Size | Q4_K_S | 9.9% smaller than Q4_K_HIFI, 2.8% smaller than Q4_K_M |
Key Insight
Without imatrix, Q4_K_HIFI provides marginal benefit over Q4_K_M. At 0.6B scale:
- 0.1% lower perplexity than Q4_K_M (23.66 vs 23.69) — essentially equal
- 3.6% lower perplexity than Q4_K_S (23.66 vs 24.55)
- 1.7% slower than Q4_K_M, 3.0% slower than Q4_K_S
- All variants lose 8-12% precision vs F16 (21.89 baseline)
However, with imatrix, Q4_K_M actually beats Q4_K_HIFI:
- Q4_K_M (imatrix): 22.92 PPL (+4.7% vs F16)
- Q4_K_HIFI (imatrix): 22.95 PPL (+4.8% vs F16)
The INT8 residual format (Q5_K_HIFI_RES8) efficiently preserves outlier precision with moderate overhead. However, imatrix quantization provides similar benefits through a different mechanism.
💡 Key Finding: For 0.6B models, Q4_K_M + imatrix is the recommended approach — it achieves slightly better quality than Q4_K_HIFI while being smaller and faster.
Precision Loss Summary (vs F16 Baseline: PPL 21.8916)
| Model | PPL (no imatrix) | vs F16 | PPL (imatrix) | vs F16 |
|---|---|---|---|---|
| Q4_K_HIFI | 23.6556 | +8.1% | 22.9451 | +4.8% |
| Q4_K_M | 23.6856 | +8.2% | 22.9210 | +4.7% ✅ |
| Q4_K_S | 24.5475 | +12.1% | 23.3136 | +6.5% |
Key Observations:
- Without imatrix: All Q4_K variants lose 8-12% precision vs F16
- With imatrix: Precision loss drops to 4.7-6.5% — imatrix recovers ~3-5.5% of lost precision
- Q4_K_M (imatrix) achieves the lowest precision loss at only +4.7% vs F16
- Q4_K_S suffers the most, with +12.1% precision loss without imatrix
Tensor Distribution
| Model | q4_K | q5_K | q6_K | Q5_K_HIFI_RES8 | f32 | Total |
|---|---|---|---|---|---|---|
| Q4_K_S | 190 | 7 | 1 | 0 | 113 | 311 |
| Q4_K_M | 169 | 0 | 29 | 0 | 113 | 311 |
| Q4_K_HIFI | 158 | 0 | 23 | 17 | 113 | 311 |
Q4_K_HIFI Enhancement: 17 critical tensors (output.weight, token_embd, attn_v layers) use Q5_K_HIFI_RES8 format with INT8 residuals + per-block scale for maximum precision.
Addendum: Impact of imatrix on ALL Quantization Types
All three quantization types can now use imatrix. Here's how they compare:
imatrix Perplexity Improvements
| Model | Without imatrix | With imatrix | Improvement | PPL vs F16 (imatrix) |
|---|---|---|---|---|
| Q4_K_HIFI | 23.6556 | 22.9451 | -0.71 (3.0% better) | +4.8% |
| Q4_K_M | 23.6856 | 22.9210 | -0.76 (3.2% better) | +4.7% ✅ |
| Q4_K_S | 24.5475 | 23.3136 | -1.23 (5.0% better) | +6.5% |
Revised Comparison (All with imatrix)
| Model | PPL (imatrix) | vs Q4_K_HIFI | vs F16 | Size |
|---|---|---|---|---|
| Q4_K_M | 22.9210 | -0.024 (-0.1%) ✅ | +4.7% ✅ | 456.11 MiB |
| Q4_K_HIFI | 22.9451 | baseline | +4.8% | 487.39 MiB |
| Q4_K_S | 23.3136 | +0.369 (+1.6%) | +6.5% | 443.30 MiB |
Key Findings
With imatrix, Q4_K_M beats Q4_K_HIFI:
| Comparison | Without imatrix | With imatrix |
|---|---|---|
| Q4_K_HIFI vs Q4_K_M | -0.1% (essentially equal) | +0.1% (Q4_K_M wins!) |
| Q4_K_HIFI vs Q4_K_S | -3.6% (moderate advantage) | -1.6% (small advantage) |
Revised Recommendations (When Using imatrix)
| Priority | Without imatrix | With imatrix |
|---|---|---|
| Quality First | Q4_K_M ≈ Q4_K_HIFI | Q4_K_M ✅ (best PPL!) |
| Best Balance | Q4_K_M | Q4_K_M ✅ |
| Size/Speed | Q4_K_S | Q4_K_S |
Conclusion
If you're using imatrix quantization:
- Q4_K_M (imatrix) is the winner — 0.1% better quality than Q4_K_HIFI (imatrix)
- Q4_K_M is 6.9% smaller and 1.7% faster than Q4_K_HIFI
- The size/speed overhead of Q4_K_HIFI provides no quality benefit with imatrix
- Q4_K_M + imatrix is the recommended choice for 0.6B models
If you're NOT using imatrix:
- Q4_K_HIFI and Q4_K_M are essentially equal (0.1% difference)
- Q4_K_M is smaller and faster, making it the practical choice
- Q4_K_M without imatrix is recommended (simpler, smaller, faster, equal quality)
Bottom line for 0.6B models: Q4_K_M (with or without imatrix) is the best choice. Q4_K_HIFI's INT8 residual format doesn't provide meaningful benefits at this model scale.
Appendix (Test Environment Details)
| Component | Specification |
|---|---|
| OS | Ubuntu 24.04.3 LTS |
| CPU | AMD EPYC 9254 24-Core Processor |
| CPU Cores | 96 cores (2 threads/core) |
| RAM | 1.0Ti |
| GPU | NVIDIA L40S × 2 |
| VRAM | 46068 MiB per GPU |
| CUDA | 12.9 |
| Test Data | wikitext-2-raw, 584 chunks |
| Context | 512 tokens |
| Samples | 200 per speed benchmark |
| imatrix | mixed-imatrix-dataset.txt, 4697 chunks |