# Qwen3-0.6B Quantization Comparison Summary

## F16 Baseline Reference
| Metric | Value |
|--------|-------|
| **F16 Perplexity** | 21.8916 |
| **File Size** | 1.40 GiB (16.00 BPW) |

All precision loss percentages below are calculated relative to this F16 baseline.

---

## Q4_K_HIFI (INT8 Residuals + Per-Block Scale)
**Pros:**
- 🏆 **Best quality without imatrix** with perplexity of 23.66 (0.1% better than Q4_K_M, 3.6% better than Q4_K_S)
- 📊 **+8.1% PPL vs F16** — moderate precision loss from quantization
- 🎯 Uses intelligent outlier preservation with INT8 residuals on critical tensors
- 🔬 17 tensors use Q5_K_HIFI_RES8 format for maximum precision on sensitive weights

**Cons:**
- 💾 **Largest file size** at 487.39 MiB (+6.9% vs Q4_K_M)
- 🐢 Slower than both Q4_K variants (1.7% slower than Q4_K_M, 3.0% slower than Q4_K_S)

**Best for:** Production deployments where output quality matters, tasks requiring accuracy (reasoning, coding, complex instructions), especially on smaller models where quantization error has larger impact.

## Performance Comparison (Q4_K_HIFI vs the others)

### Q4_K_M

| Metric              | Q4_K_HIFI  | Q4_K_M     | Difference                    |
|---------------------|------------|------------|-------------------------------|
| **Speed (TPS)**     | 614.32     | 624.66     | -10.34 (1.7% slower)          |
| **Perplexity**      | 23.6556    | 23.6856    | **-0.03 (0.1% better)**       |
| **PPL vs F16**      | +8.1%      | +8.2%      | 0.1% less precision loss      |
| **File Size**       | 487.39 MiB | 456.11 MiB | +31.28 MiB (6.9% larger)      |
| **Bits Per Weight** | 5.44       | 5.09       | +0.35 (6.9% more)             |

**Pros:**
- ⚖️ Traditional "balanced" approach between speed and quality
- 📚 Well-documented, standard quantization method
- 💾 Smaller file size than Q4_K_HIFI
- ⚡ 1.7% faster inference than Q4_K_HIFI
- 📊 **+8.2% PPL vs F16** — nearly identical precision loss to Q4_K_HIFI

**Cons:**
- ❌ **Slightly lower quality** (essentially equal - only 0.1% higher perplexity than Q4_K_HIFI)

**Best for:** When storage is constrained but you still need reasonable quality.
**Summary:** Without imatrix, Q4_K_HIFI and Q4_K_M are essentially identical in quality (0.1% difference). Q4_K_M is smaller (6.9% less) and faster (1.7%), making it the better choice without imatrix.

### Q4_K_S

| Metric              | Q4_K_HIFI  | Q4_K_S     | Difference                    |
|---------------------|------------|------------|-------------------------------|
| **Speed (TPS)**     | 614.32     | 632.79     | -18.47 (3.0% slower)          |
| **Perplexity**      | 23.6556    | 24.5475    | **-0.89 (3.6% better)**       |
| **PPL vs F16**      | +8.1%      | +12.1%     | 4.0% less precision loss      |
| **File Size**       | 487.39 MiB | 443.30 MiB | +44.09 MiB (9.9% larger)      |
| **Bits Per Weight** | 5.44       | 4.95       | +0.49 (9.9% more)             |

**Pros:**
- ⚡ **Fastest inference** at 632.79 TPS (3.0% faster than Q4_K_HIFI)
- 💾 **Smallest file size** at 443.30 MiB
- ✅ Best choice when speed and storage are critical

**Cons:**
- ❌ **Significantly worse quality** with perplexity of 24.55 (3.6% higher than Q4_K_HIFI)
- 📊 **+12.1% PPL vs F16** — highest precision loss of all Q4_K variants
- ⚠️ Uses minimal q6_K enhancement — impacts quality on sensitive weights

**Best for:** Extreme resource constraints, quick prototyping, or bulk processing where quality is less important.
**Summary:** Q4_K_HIFI trades a 3.0% speed reduction and 9.9% larger file size for a 3.6% improvement in quality.

---

## Recommendation Matrix (Without imatrix)

| Priority          | Recommended Model | Rationale                                                                      |
|-------------------|-------------------|--------------------------------------------------------------------------------|
| **Quality First** | Q4_K_HIFI         | 3.6% better perplexity than Q4_K_S, essentially equal to Q4_K_M                |
| **Speed First**   | Q4_K_S            | 3.0% faster, acceptable if quality degradation is tolerable                    |
| **Best Balance**  | Q4_K_M            | Equal quality to Q4_K_HIFI, smaller (6.9%) and faster (1.7%)                   |
| **Smallest Size** | Q4_K_S            | 9.9% smaller than Q4_K_HIFI, 2.8% smaller than Q4_K_M                          |

---

## Key Insight

**Without imatrix, Q4_K_HIFI provides marginal benefit over Q4_K_M.** At 0.6B scale:
- **0.1% lower perplexity** than Q4_K_M (23.66 vs 23.69) — essentially equal
- **3.6% lower perplexity** than Q4_K_S (23.66 vs 24.55)
- **1.7% slower** than Q4_K_M, **3.0% slower** than Q4_K_S
- **All variants lose 8-12% precision vs F16** (21.89 baseline)

**However, with imatrix, Q4_K_M actually beats Q4_K_HIFI:**
- Q4_K_M (imatrix): **22.92** PPL (+4.7% vs F16)
- Q4_K_HIFI (imatrix): **22.95** PPL (+4.8% vs F16)

The INT8 residual format (Q5_K_HIFI_RES8) efficiently preserves outlier precision with moderate overhead. However, imatrix quantization provides similar benefits through a different mechanism.

💡 **Key Finding:** For 0.6B models, **Q4_K_M + imatrix** is the recommended approach — it achieves slightly better quality than Q4_K_HIFI while being smaller and faster.

---

## Precision Loss Summary (vs F16 Baseline: PPL 21.8916)

| Model     | PPL (no imatrix) | vs F16 | PPL (imatrix) | vs F16 |
|-----------|------------------|--------|---------------|--------|
| Q4_K_HIFI | 23.6556          | **+8.1%** | 22.9451 | **+4.8%** |
| Q4_K_M    | 23.6856          | **+8.2%** | 22.9210 | **+4.7%** ✅ |
| Q4_K_S    | 24.5475          | **+12.1%** | 23.3136 | **+6.5%** |

**Key Observations:**
- Without imatrix: All Q4_K variants lose 8-12% precision vs F16
- With imatrix: Precision loss drops to 4.7-6.5% — imatrix recovers ~3-5.5% of lost precision
- Q4_K_M (imatrix) achieves the **lowest precision loss** at only +4.7% vs F16
- Q4_K_S suffers the most, with +12.1% precision loss without imatrix

---

## Tensor Distribution

| Model     | q4_K | q5_K | q6_K | Q5_K_HIFI_RES8 | f32 | Total |
|-----------|------|------|------|----------------|-----|-------|
| Q4_K_S    | 190  | 7    | 1    | 0              | 113 | 311   |
| Q4_K_M    | 169  | 0    | 29   | 0              | 113 | 311   |
| Q4_K_HIFI | 158  | 0    | 23   | 17             | 113 | 311   |

**Q4_K_HIFI Enhancement:** 17 critical tensors (output.weight, token_embd, attn_v layers) use Q5_K_HIFI_RES8 format with INT8 residuals + per-block scale for maximum precision.

---

## Addendum: Impact of imatrix on ALL Quantization Types

All three quantization types can now use imatrix. Here's how they compare:

### imatrix Perplexity Improvements

| Model     | Without imatrix | With imatrix | Improvement | PPL vs F16 (imatrix) |
|-----------|-----------------|--------------|-------------|----------------------|
| Q4_K_HIFI | 23.6556         | **22.9451**  | **-0.71 (3.0% better)** | **+4.8%** |
| Q4_K_M    | 23.6856         | **22.9210**  | **-0.76 (3.2% better)** | **+4.7%** ✅ |
| Q4_K_S    | 24.5475         | **23.3136**  | **-1.23 (5.0% better)** | **+6.5%** |

### Revised Comparison (All with imatrix)

| Model     | PPL (imatrix) | vs Q4_K_HIFI | vs F16 | Size |
|-----------|---------------|--------------|--------|------|
| Q4_K_M    | **22.9210**   | **-0.024 (-0.1%)** ✅ | **+4.7%** ✅ | 456.11 MiB |
| Q4_K_HIFI | 22.9451       | baseline   | +4.8% | 487.39 MiB |
| Q4_K_S    | 23.3136       | +0.369 (+1.6%)  | +6.5% | 443.30 MiB |

### Key Findings

**With imatrix, Q4_K_M beats Q4_K_HIFI:**

| Comparison | Without imatrix | With imatrix |
|------------|-----------------|--------------|
| Q4_K_HIFI vs Q4_K_M | **-0.1%** (essentially equal) | **+0.1%** (Q4_K_M wins!) |
| Q4_K_HIFI vs Q4_K_S | **-3.6%** (moderate advantage) | **-1.6%** (small advantage) |

### Revised Recommendations (When Using imatrix)

| Priority          | Without imatrix | With imatrix |
|-------------------|-----------------|--------------|
| **Quality First** | Q4_K_M ≈ Q4_K_HIFI | **Q4_K_M** ✅ (best PPL!) |
| **Best Balance**  | Q4_K_M | **Q4_K_M** ✅ |
| **Size/Speed**    | Q4_K_S | Q4_K_S |

### Conclusion

**If you're using imatrix quantization:**
- **Q4_K_M (imatrix) is the winner** — 0.1% better quality than Q4_K_HIFI (imatrix)
- Q4_K_M is 6.9% smaller and 1.7% faster than Q4_K_HIFI
- The size/speed overhead of Q4_K_HIFI provides **no quality benefit** with imatrix
- **Q4_K_M + imatrix is the recommended choice for 0.6B models**

**If you're NOT using imatrix:**
- Q4_K_HIFI and Q4_K_M are essentially equal (0.1% difference)
- Q4_K_M is smaller and faster, making it the practical choice
- **Q4_K_M without imatrix is recommended** (simpler, smaller, faster, equal quality)

**Bottom line for 0.6B models:** Q4_K_M (with or without imatrix) is the best choice. Q4_K_HIFI's INT8 residual format doesn't provide meaningful benefits at this model scale.

---

## Appendix (Test Environment Details)

| Component     | Specification                          |
|---------------|----------------------------------------|
| **OS**        | Ubuntu 24.04.3 LTS                     |
| **CPU**       | AMD EPYC 9254 24-Core Processor        |
| **CPU Cores** | 96 cores (2 threads/core)              |
| **RAM**       | 1.0Ti                                  |
| **GPU**       | NVIDIA L40S × 2                        |
| **VRAM**      | 46068 MiB per GPU                      |
| **CUDA**      | 12.9                                   |
| **Test Data** | wikitext-2-raw, 584 chunks             |
| **Context**   | 512 tokens                             |
| **Samples**   | 200 per speed benchmark                |
| **imatrix**   | mixed-imatrix-dataset.txt, 4697 chunks |