# Qwen3-0.6B Quantization Comparison Summary

## Q3_HIFI (Adaptive/Custom)
**Pros:**
- 🏆 **Best quality** with lowest perplexity of 26.43 (16.4% better than Q3_K_M, 26.0% better than Q3_K_S)
- 📦 **Smaller than Q3_K_M** (382.37 vs 389.12 MiB) while being significantly better quality
- 🎯 Uses intelligent layer-sensitive quantization (Q3_HIFI on sensitive layers, mixed q3_K/q4_K elsewhere)
- 📊 Most consistent results (lowest standard deviation in perplexity: ±0.23)

**Cons:**
- 🐢 **Slowest inference** at 601.4 TPS (2.8% slower than Q3_K_M)
- 🔧 Custom quantization may have less community support

**Best for:** Production deployments where output quality matters, tasks requiring accuracy (reasoning, coding, complex instructions), or when you want the best quality-to-size ratio.

## Performance Comparison (Q3_HIFI vs the others)

### Q3_K_M

| Metric              | Q3_HIFI   | Q3_K_M    | Difference                    |
|---------------------|-----------|-----------|-------------------------------|
| **Speed (TPS)**     | 601.39    | 618.42    | -17.03 (2.8% slower)          |
| **Perplexity**      | 26.43     | 31.64     | **-5.21 (16.4% better)**      |
| **File Size**       | 382.37 MiB| 389.12 MiB| **-6.75 MiB (1.7% smaller)**  |
| **Bits Per Weight** | 4.27      | 4.34      | -0.07 (1.6% less)             |

**Pros:**
- ⚖️ Traditional "balanced" approach between speed and quality
- 📚 Well-documented, standard quantization method
- ⚡ Fastest inference speed (618.4 TPS)

**Cons:**
- 💾 **Largest file size** at 389.12 MiB despite not being the best quality
- ❌ **Outclassed by Q3_HIFI** which is smaller AND better quality

**Best for:** Legacy compatibility or when you need a proven, standard quantization approach with maximum throughput.
**Summary:** Q3_HIFI delivers significantly better quality (16.4% lower perplexity) in a smaller package (1.7% less storage) with only a marginal 2.8% speed penalty.

### Q3_K_S

| Metric              | Q3_HIFI   | Q3_K_S    | Difference                |
|---------------------|-----------|-----------|---------------------------|
| **Speed (TPS)**     | 601.39    | 612.28    | -10.89 (1.8% slower)      |
| **Perplexity**      | 26.43     | 35.70     | **-9.27 (26.0% better)**  |
| **File Size**       | 382.37 MiB| 366.19 MiB| +16.18 MiB (4.4% larger)  |
| **Bits Per Weight** | 4.27      | 4.09      | +0.18 (4.4% more)         |

**Pros:**
- 💾 **Smallest file size** at 366.19 MiB
- ✅ Best choice when storage is the absolute limiting factor

**Cons:**
- ❌ **Worst quality** with perplexity of 35.70 (35% higher than Q3_HIFI)
- 🐌 Not actually faster than Q3_K_M (612 vs 618 TPS)
- Uses only q3_K quantization throughout (no mixed precision)

**Best for:** Extremely memory-constrained environments where every megabyte counts.
**Summary:** Q3_HIFI trades a 1.8% speed reduction and 4.4% larger file size for a substantial 26.0% improvement in quality (lower perplexity).

---

## Recommendation Matrix

| Priority          | Recommended Model | Rationale                                                                    |
|-------------------|-------------------|------------------------------------------------------------------------------|
| **Quality First** | Q3_HIFI           | 26% better perplexity than Q3_K_S with minimal speed loss                    |
| **Speed First**   | Q3_K_M            | Fastest inference at 618 TPS, acceptable quality tradeoff                    |
| **Best Balance**  | Q3_HIFI           | Better quality AND smaller size than Q3_K_M, only 2.8% slower                |
| **Smallest Size** | Q3_K_S            | 4% smaller than Q3_HIFI, 6% smaller than Q3_K_M                              |

---

## Key Insight

**Q3_HIFI represents a clear advancement** over the traditional Q3_K_M approach. It achieves:
- **16.4% lower perplexity** (better accuracy)
- **1.7% smaller file size** (382 vs 389 MiB)
- Only **2.8% slower** inference (601 vs 618 TPS)

The Q3_K_M quantization is essentially obsoleted by Q3_HIFI for most use cases. Q3_K_S offers marginal storage savings but with significantly degraded quality and is actually slower than Q3_K_M, making it difficult to recommend.

The only remaining practical choice is between **Q3_K_M** (maximum speed) and **Q3_HIFI** (maximum quality). Given the 2.8% speed difference is imperceptible at 600+ TPS, **Q3_HIFI is the recommended default**.

## Appendix (Test Environment Details)

| Component     | Specification                   |
|---------------|---------------------------------|
| **OS**        | Ubuntu 24.04.3 LTS              |
| **CPU**       | AMD EPYC 9254 24-Core Processor |
| **CPU Cores** | 96 cores (2 threads/core)       |
| **RAM**       | 1.0Ti                           |
| **GPU**       | NVIDIA L40S × 2                 |
| **VRAM**      | 46068 MiB per GPU               |
| **CUDA**      | 12.9                            |