95 lines
4.8 KiB
Markdown
95 lines
4.8 KiB
Markdown
|
# Qwen3-0.6B Quantization Comparison Summary
|
|||
|
|
|
|||
|
|
## Q3_HIFI (Adaptive/Custom)
|
|||
|
|
**Pros:**
|
|||
|
|
- 🏆 **Best quality** with lowest perplexity of 26.43 (16.4% better than Q3_K_M, 26.0% better than Q3_K_S)
|
|||
|
|
- 📦 **Smaller than Q3_K_M** (382.37 vs 389.12 MiB) while being significantly better quality
|
|||
|
|
- 🎯 Uses intelligent layer-sensitive quantization (Q3_HIFI on sensitive layers, mixed q3_K/q4_K elsewhere)
|
|||
|
|
- 📊 Most consistent results (lowest standard deviation in perplexity: ±0.23)
|
|||
|
|
|
|||
|
|
**Cons:**
|
|||
|
|
- 🐢 **Slowest inference** at 601.4 TPS (2.8% slower than Q3_K_M)
|
|||
|
|
- 🔧 Custom quantization may have less community support
|
|||
|
|
|
|||
|
|
**Best for:** Production deployments where output quality matters, tasks requiring accuracy (reasoning, coding, complex instructions), or when you want the best quality-to-size ratio.
|
|||
|
|
|
|||
|
|
## Performance Comparison (Q3_HIFI vs the others)
|
|||
|
|
|
|||
|
|
### Q3_K_M
|
|||
|
|
|
|||
|
|
| Metric | Q3_HIFI | Q3_K_M | Difference |
|
|||
|
|
|---------------------|-----------|-----------|-------------------------------|
|
|||
|
|
| **Speed (TPS)** | 601.39 | 618.42 | -17.03 (2.8% slower) |
|
|||
|
|
| **Perplexity** | 26.43 | 31.64 | **-5.21 (16.4% better)** |
|
|||
|
|
| **File Size** | 382.37 MiB| 389.12 MiB| **-6.75 MiB (1.7% smaller)** |
|
|||
|
|
| **Bits Per Weight** | 4.27 | 4.34 | -0.07 (1.6% less) |
|
|||
|
|
|
|||
|
|
**Pros:**
|
|||
|
|
- ⚖️ Traditional "balanced" approach between speed and quality
|
|||
|
|
- 📚 Well-documented, standard quantization method
|
|||
|
|
- ⚡ Fastest inference speed (618.4 TPS)
|
|||
|
|
|
|||
|
|
**Cons:**
|
|||
|
|
- 💾 **Largest file size** at 389.12 MiB despite not being the best quality
|
|||
|
|
- ❌ **Outclassed by Q3_HIFI** which is smaller AND better quality
|
|||
|
|
|
|||
|
|
**Best for:** Legacy compatibility or when you need a proven, standard quantization approach with maximum throughput.
|
|||
|
|
**Summary:** Q3_HIFI delivers significantly better quality (16.4% lower perplexity) in a smaller package (1.7% less storage) with only a marginal 2.8% speed penalty.
|
|||
|
|
|
|||
|
|
### Q3_K_S
|
|||
|
|
|
|||
|
|
| Metric | Q3_HIFI | Q3_K_S | Difference |
|
|||
|
|
|---------------------|-----------|-----------|---------------------------|
|
|||
|
|
| **Speed (TPS)** | 601.39 | 612.28 | -10.89 (1.8% slower) |
|
|||
|
|
| **Perplexity** | 26.43 | 35.70 | **-9.27 (26.0% better)** |
|
|||
|
|
| **File Size** | 382.37 MiB| 366.19 MiB| +16.18 MiB (4.4% larger) |
|
|||
|
|
| **Bits Per Weight** | 4.27 | 4.09 | +0.18 (4.4% more) |
|
|||
|
|
|
|||
|
|
**Pros:**
|
|||
|
|
- 💾 **Smallest file size** at 366.19 MiB
|
|||
|
|
- ✅ Best choice when storage is the absolute limiting factor
|
|||
|
|
|
|||
|
|
**Cons:**
|
|||
|
|
- ❌ **Worst quality** with perplexity of 35.70 (35% higher than Q3_HIFI)
|
|||
|
|
- 🐌 Not actually faster than Q3_K_M (612 vs 618 TPS)
|
|||
|
|
- Uses only q3_K quantization throughout (no mixed precision)
|
|||
|
|
|
|||
|
|
**Best for:** Extremely memory-constrained environments where every megabyte counts.
|
|||
|
|
**Summary:** Q3_HIFI trades a 1.8% speed reduction and 4.4% larger file size for a substantial 26.0% improvement in quality (lower perplexity).
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Recommendation Matrix
|
|||
|
|
|
|||
|
|
| Priority | Recommended Model | Rationale |
|
|||
|
|
|-------------------|-------------------|------------------------------------------------------------------------------|
|
|||
|
|
| **Quality First** | Q3_HIFI | 26% better perplexity than Q3_K_S with minimal speed loss |
|
|||
|
|
| **Speed First** | Q3_K_M | Fastest inference at 618 TPS, acceptable quality tradeoff |
|
|||
|
|
| **Best Balance** | Q3_HIFI | Better quality AND smaller size than Q3_K_M, only 2.8% slower |
|
|||
|
|
| **Smallest Size** | Q3_K_S | 4% smaller than Q3_HIFI, 6% smaller than Q3_K_M |
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Key Insight
|
|||
|
|
|
|||
|
|
**Q3_HIFI represents a clear advancement** over the traditional Q3_K_M approach. It achieves:
|
|||
|
|
- **16.4% lower perplexity** (better accuracy)
|
|||
|
|
- **1.7% smaller file size** (382 vs 389 MiB)
|
|||
|
|
- Only **2.8% slower** inference (601 vs 618 TPS)
|
|||
|
|
|
|||
|
|
The Q3_K_M quantization is essentially obsoleted by Q3_HIFI for most use cases. Q3_K_S offers marginal storage savings but with significantly degraded quality and is actually slower than Q3_K_M, making it difficult to recommend.
|
|||
|
|
|
|||
|
|
The only remaining practical choice is between **Q3_K_M** (maximum speed) and **Q3_HIFI** (maximum quality). Given the 2.8% speed difference is imperceptible at 600+ TPS, **Q3_HIFI is the recommended default**.
|
|||
|
|
|
|||
|
|
## Appendix (Test Environment Details)
|
|||
|
|
|
|||
|
|
| Component | Specification |
|
|||
|
|
|---------------|---------------------------------|
|
|||
|
|
| **OS** | Ubuntu 24.04.3 LTS |
|
|||
|
|
| **CPU** | AMD EPYC 9254 24-Core Processor |
|
|||
|
|
| **CPU Cores** | 96 cores (2 threads/core) |
|
|||
|
|
| **RAM** | 1.0Ti |
|
|||
|
|
| **GPU** | NVIDIA L40S × 2 |
|
|||
|
|
| **VRAM** | 46068 MiB per GPU |
|
|||
|
|
| **CUDA** | 12.9 |
|