92 lines
4.7 KiB
Markdown
92 lines
4.7 KiB
Markdown
# Qwen3-8B Quantization Comparison Summary
|
||
|
||
## Q3_HIFI (Adaptive/Custom)
|
||
**Pros:**
|
||
- 🏆 **Best quality** with lowest perplexity of 10.56 (4.4% better than Q3_K_M, 7.2% better than Q3_K_S)
|
||
- 📦 **Smaller than Q3_K_M** (3.72 vs 3.84 GiB) while being significantly better quality
|
||
- 🎯 Uses intelligent layer-sensitive quantization (Q3_HIFI on sensitive layers, mixed q3_K/q4_K elsewhere)
|
||
- 📊 Most consistent results (lowest relative standard deviation in perplexity)
|
||
|
||
**Cons:**
|
||
- 🐢 **Slowest inference** at 143.98 TPS (6.3% slower than Q3_K_S)
|
||
- 🔧 Custom quantization may have less community support
|
||
|
||
**Best for:** Production deployments where output quality matters, tasks requiring accuracy (reasoning, coding, complex instructions), or when you want the best quality-to-size ratio.
|
||
|
||
## Performance Comparison (Q3_HIFI vs the others)
|
||
|
||
### Q3_K_M
|
||
|
||
| Metric | Q3_HIFI | Q3_K_M | Difference |
|
||
|---------------------|----------|----------|------------------------------|
|
||
| **Speed (TPS)** | 143.98 | 144.72 | -0.74 (0.5% slower) |
|
||
| **Perplexity** | 10.56 | 11.05 | **-0.49 (4.4% better)** |
|
||
| **File Size** | 3.72 GiB | 3.84 GiB | **-0.12 GiB (3.1% smaller)** |
|
||
| **Bits Per Weight** | 3.90 | 4.02 | -0.12 (3.0% less) |
|
||
|
||
**Pros:**
|
||
- ⚖️ Traditional "balanced" approach between speed and quality
|
||
- 📚 Well-documented, standard quantization method
|
||
|
||
**Cons:**
|
||
- 💾 **Largest file size** at 3.84 GiB despite not being the best quality
|
||
- 🐌 Middle-of-the-road speed (144.7 TPS)
|
||
- ❌ **Outclassed by Q3_HIFI** which is smaller AND better quality
|
||
|
||
**Best for:** Legacy compatibility or when you need a proven, standard quantization approach.
|
||
**Summary:** Q3_HIFI delivers significantly better quality (4.4% lower perplexity) in a smaller package (3.1% less storage) with virtually no speed penalty (0.5% slower).
|
||
|
||
### Q3_K_S
|
||
|
||
| Metric | Q3_HIFI | Q3_K_S | Difference |
|
||
|---------------------|----------|----------|-------------------------|
|
||
| **Speed (TPS)** | 143.98 | 153.74 | -9.76 (6.3% slower) |
|
||
| **Perplexity** | 10.56 | 11.38 | **-0.82 (7.2% better)** |
|
||
| **File Size** | 3.72 GiB | 3.51 GiB | +0.21 GiB (6.0% larger) |
|
||
| **Bits Per Weight** | 3.90 | 3.68 | +0.22 (6.0% more) |
|
||
|
||
**Pros:**
|
||
- ⚡ **Fastest inference** at 153.74 TPS (~6% faster than Q3_K_M, ~7% faster than Q3_HIFI)
|
||
- 💾 **Smallest file size** at 3.51 GiB
|
||
- ✅ Best choice when speed and storage are critical
|
||
|
||
**Cons:**
|
||
- ❌ **Worst quality** with perplexity of 11.38 (7.2% higher than Q3_HIFI)
|
||
- Uses only q3_K quantization throughout (no mixed precision)
|
||
|
||
**Best for:** Resource-constrained environments, real-time applications where latency matters more than accuracy, or initial prototyping.
|
||
**Summary:** Q3_HIFI trades a 6.3% speed reduction and 6.0% larger file size for a substantial 7.2% improvement in quality (lower perplexity).
|
||
|
||
---
|
||
|
||
## Recommendation Matrix
|
||
|
||
| Priority | Recommended Model | Rationale |
|
||
|-------------------|-------------------|-----------------------------------------------------------------------------|
|
||
| **Quality First** | Q3_HIFI | 7.2% better perplexity than Q3_K_S with acceptable speed loss |
|
||
| **Speed First** | Q3_K_S | 6.3% faster inference, acceptable quality tradeoff for latency-sensitive apps |
|
||
| **Best Balance** | Q3_HIFI | Better quality AND smaller size than Q3_K_M, only 0.5% slower |
|
||
| **Smallest Size** | Q3_K_S | 6% smaller than alternatives |
|
||
|
||
---
|
||
|
||
## Key Insight
|
||
|
||
**Q3_HIFI represents a clear advancement** over the traditional Q3_K_M approach. It achieves:
|
||
- **4.4% lower perplexity** (better accuracy)
|
||
- **3.1% smaller file size** (3.72 vs 3.84 GiB)
|
||
- Only **0.5% slower** inference (144.0 vs 144.7 TPS)
|
||
|
||
The Q3_K_M quantization is essentially obsoleted by Q3_HIFI for most use cases. The only remaining choice is between **Q3_K_S** (maximum speed, acceptable quality) and **Q3_HIFI** (maximum quality, acceptable speed).
|
||
|
||
## Appendix (Test Environment Details)
|
||
|
||
| Component | Specification |
|
||
|---------------|---------------------------------|
|
||
| **OS** | Ubuntu 24.04.3 LTS |
|
||
| **CPU** | AMD EPYC 9254 24-Core Processor |
|
||
| **CPU Cores** | 96 cores (2 threads/core) |
|
||
| **RAM** | 1.0Ti |
|
||
| **GPU** | NVIDIA L40S × 2 |
|
||
| **VRAM** | 46068 MiB per GPU |
|
||
| **CUDA** | 12.9 | |