Qwen3-8B-f16/Q3_Quantization_Comparison.md

# Qwen3-8B Quantization Comparison Summary

## Q3_HIFI (Adaptive/Custom)
**Pros:**
- 🏆 **Best quality** with lowest perplexity of 10.56 (4.4% better than Q3_K_M, 7.2% better than Q3_K_S)
- 📦 **Smaller than Q3_K_M** (3.72 vs 3.84 GiB) while being significantly better quality
- 🎯 Uses intelligent layer-sensitive quantization (Q3_HIFI on sensitive layers, mixed q3_K/q4_K elsewhere)
- 📊 Most consistent results (lowest relative standard deviation in perplexity)

**Cons:**
- 🐢 **Slowest inference** at 143.98 TPS (6.3% slower than Q3_K_S)
- 🔧 Custom quantization may have less community support

**Best for:** Production deployments where output quality matters, tasks requiring accuracy (reasoning, coding, complex instructions), or when you want the best quality-to-size ratio.

## Performance Comparison (Q3_HIFI vs the others)

### Q3_K_M

| Metric              | Q3_HIFI  | Q3_K_M   | Difference                   |
|---------------------|----------|----------|------------------------------|
| **Speed (TPS)**     | 143.98   | 144.72   | -0.74 (0.5% slower)          |
| **Perplexity**      | 10.56    | 11.05    | **-0.49 (4.4% better)**      |
| **File Size**       | 3.72 GiB | 3.84 GiB | **-0.12 GiB (3.1% smaller)** |
| **Bits Per Weight** | 3.90     | 4.02     | -0.12 (3.0% less)            |

**Pros:**
- ⚖️ Traditional "balanced" approach between speed and quality
- 📚 Well-documented, standard quantization method

**Cons:**
- 💾 **Largest file size** at 3.84 GiB despite not being the best quality
- 🐌 Middle-of-the-road speed (144.7 TPS)
- ❌ **Outclassed by Q3_HIFI** which is smaller AND better quality

**Best for:** Legacy compatibility or when you need a proven, standard quantization approach.
**Summary:** Q3_HIFI delivers significantly better quality (4.4% lower perplexity) in a smaller package (3.1% less storage) with virtually no speed penalty (0.5% slower).

### Q3_K_S

| Metric              | Q3_HIFI  | Q3_K_S   | Difference              |
|---------------------|----------|----------|-------------------------|
| **Speed (TPS)**     | 143.98   | 153.74   | -9.76 (6.3% slower)     |
| **Perplexity**      | 10.56    | 11.38    | **-0.82 (7.2% better)** |
| **File Size**       | 3.72 GiB | 3.51 GiB | +0.21 GiB (6.0% larger) |
| **Bits Per Weight** | 3.90     | 3.68     | +0.22 (6.0% more)       |

**Pros:**
- ⚡ **Fastest inference** at 153.74 TPS (~6% faster than Q3_K_M, ~7% faster than Q3_HIFI)
- 💾 **Smallest file size** at 3.51 GiB
- ✅ Best choice when speed and storage are critical

**Cons:**
- ❌ **Worst quality** with perplexity of 11.38 (7.2% higher than Q3_HIFI)
- Uses only q3_K quantization throughout (no mixed precision)

**Best for:** Resource-constrained environments, real-time applications where latency matters more than accuracy, or initial prototyping.
**Summary:** Q3_HIFI trades a 6.3% speed reduction and 6.0% larger file size for a substantial 7.2% improvement in quality (lower perplexity).

---

## Recommendation Matrix

| Priority          | Recommended Model | Rationale                                                                   |
|-------------------|-------------------|-----------------------------------------------------------------------------|
| **Quality First** | Q3_HIFI           | 7.2% better perplexity than Q3_K_S with acceptable speed loss               |
| **Speed First**   | Q3_K_S            | 6.3% faster inference, acceptable quality tradeoff for latency-sensitive apps |
| **Best Balance**  | Q3_HIFI           | Better quality AND smaller size than Q3_K_M, only 0.5% slower               |
| **Smallest Size** | Q3_K_S            | 6% smaller than alternatives                                                |

---

## Key Insight

**Q3_HIFI represents a clear advancement** over the traditional Q3_K_M approach. It achieves:
- **4.4% lower perplexity** (better accuracy)
- **3.1% smaller file size** (3.72 vs 3.84 GiB)
- Only **0.5% slower** inference (144.0 vs 144.7 TPS)

The Q3_K_M quantization is essentially obsoleted by Q3_HIFI for most use cases. The only remaining choice is between **Q3_K_S** (maximum speed, acceptable quality) and **Q3_HIFI** (maximum quality, acceptable speed).

## Appendix (Test Environment Details)

| Component     | Specification                   |
|---------------|---------------------------------|
| **OS**        | Ubuntu 24.04.3 LTS              |
| **CPU**       | AMD EPYC 9254 24-Core Processor |
| **CPU Cores** | 96 cores (2 threads/core)       |
| **RAM**       | 1.0Ti                           |
| **GPU**       | NVIDIA L40S × 2                 |
| **VRAM**      | 46068 MiB per GPU               |
| **CUDA**      | 12.9                            |