202 lines
9.6 KiB
Markdown
202 lines
9.6 KiB
Markdown
# Qwen3-0.6B Quantization Comparison Summary
|
||
|
||
## F16 Baseline Reference
|
||
| Metric | Value |
|
||
|--------|-------|
|
||
| **F16 Perplexity** | 21.8916 |
|
||
| **File Size** | 1.40 GiB (16.00 BPW) |
|
||
|
||
All precision loss percentages below are calculated relative to this F16 baseline.
|
||
|
||
---
|
||
|
||
## Q4_K_HIFI (INT8 Residuals + Per-Block Scale)
|
||
**Pros:**
|
||
- 🏆 **Best quality without imatrix** with perplexity of 23.66 (0.1% better than Q4_K_M, 3.6% better than Q4_K_S)
|
||
- 📊 **+8.1% PPL vs F16** — moderate precision loss from quantization
|
||
- 🎯 Uses intelligent outlier preservation with INT8 residuals on critical tensors
|
||
- 🔬 17 tensors use Q5_K_HIFI_RES8 format for maximum precision on sensitive weights
|
||
|
||
**Cons:**
|
||
- 💾 **Largest file size** at 487.39 MiB (+6.9% vs Q4_K_M)
|
||
- 🐢 Slower than both Q4_K variants (1.7% slower than Q4_K_M, 3.0% slower than Q4_K_S)
|
||
|
||
**Best for:** Production deployments where output quality matters, tasks requiring accuracy (reasoning, coding, complex instructions), especially on smaller models where quantization error has larger impact.
|
||
|
||
## Performance Comparison (Q4_K_HIFI vs the others)
|
||
|
||
### Q4_K_M
|
||
|
||
| Metric | Q4_K_HIFI | Q4_K_M | Difference |
|
||
|---------------------|------------|------------|-------------------------------|
|
||
| **Speed (TPS)** | 614.32 | 624.66 | -10.34 (1.7% slower) |
|
||
| **Perplexity** | 23.6556 | 23.6856 | **-0.03 (0.1% better)** |
|
||
| **PPL vs F16** | +8.1% | +8.2% | 0.1% less precision loss |
|
||
| **File Size** | 487.39 MiB | 456.11 MiB | +31.28 MiB (6.9% larger) |
|
||
| **Bits Per Weight** | 5.44 | 5.09 | +0.35 (6.9% more) |
|
||
|
||
**Pros:**
|
||
- ⚖️ Traditional "balanced" approach between speed and quality
|
||
- 📚 Well-documented, standard quantization method
|
||
- 💾 Smaller file size than Q4_K_HIFI
|
||
- ⚡ 1.7% faster inference than Q4_K_HIFI
|
||
- 📊 **+8.2% PPL vs F16** — nearly identical precision loss to Q4_K_HIFI
|
||
|
||
**Cons:**
|
||
- ❌ **Slightly lower quality** (essentially equal - only 0.1% higher perplexity than Q4_K_HIFI)
|
||
|
||
**Best for:** When storage is constrained but you still need reasonable quality.
|
||
**Summary:** Without imatrix, Q4_K_HIFI and Q4_K_M are essentially identical in quality (0.1% difference). Q4_K_M is smaller (6.9% less) and faster (1.7%), making it the better choice without imatrix.
|
||
|
||
### Q4_K_S
|
||
|
||
| Metric | Q4_K_HIFI | Q4_K_S | Difference |
|
||
|---------------------|------------|------------|-------------------------------|
|
||
| **Speed (TPS)** | 614.32 | 632.79 | -18.47 (3.0% slower) |
|
||
| **Perplexity** | 23.6556 | 24.5475 | **-0.89 (3.6% better)** |
|
||
| **PPL vs F16** | +8.1% | +12.1% | 4.0% less precision loss |
|
||
| **File Size** | 487.39 MiB | 443.30 MiB | +44.09 MiB (9.9% larger) |
|
||
| **Bits Per Weight** | 5.44 | 4.95 | +0.49 (9.9% more) |
|
||
|
||
**Pros:**
|
||
- ⚡ **Fastest inference** at 632.79 TPS (3.0% faster than Q4_K_HIFI)
|
||
- 💾 **Smallest file size** at 443.30 MiB
|
||
- ✅ Best choice when speed and storage are critical
|
||
|
||
**Cons:**
|
||
- ❌ **Significantly worse quality** with perplexity of 24.55 (3.6% higher than Q4_K_HIFI)
|
||
- 📊 **+12.1% PPL vs F16** — highest precision loss of all Q4_K variants
|
||
- ⚠️ Uses minimal q6_K enhancement — impacts quality on sensitive weights
|
||
|
||
**Best for:** Extreme resource constraints, quick prototyping, or bulk processing where quality is less important.
|
||
**Summary:** Q4_K_HIFI trades a 3.0% speed reduction and 9.9% larger file size for a 3.6% improvement in quality.
|
||
|
||
---
|
||
|
||
## Recommendation Matrix (Without imatrix)
|
||
|
||
| Priority | Recommended Model | Rationale |
|
||
|-------------------|-------------------|--------------------------------------------------------------------------------|
|
||
| **Quality First** | Q4_K_HIFI | 3.6% better perplexity than Q4_K_S, essentially equal to Q4_K_M |
|
||
| **Speed First** | Q4_K_S | 3.0% faster, acceptable if quality degradation is tolerable |
|
||
| **Best Balance** | Q4_K_M | Equal quality to Q4_K_HIFI, smaller (6.9%) and faster (1.7%) |
|
||
| **Smallest Size** | Q4_K_S | 9.9% smaller than Q4_K_HIFI, 2.8% smaller than Q4_K_M |
|
||
|
||
---
|
||
|
||
## Key Insight
|
||
|
||
**Without imatrix, Q4_K_HIFI provides marginal benefit over Q4_K_M.** At 0.6B scale:
|
||
- **0.1% lower perplexity** than Q4_K_M (23.66 vs 23.69) — essentially equal
|
||
- **3.6% lower perplexity** than Q4_K_S (23.66 vs 24.55)
|
||
- **1.7% slower** than Q4_K_M, **3.0% slower** than Q4_K_S
|
||
- **All variants lose 8-12% precision vs F16** (21.89 baseline)
|
||
|
||
**However, with imatrix, Q4_K_M actually beats Q4_K_HIFI:**
|
||
- Q4_K_M (imatrix): **22.92** PPL (+4.7% vs F16)
|
||
- Q4_K_HIFI (imatrix): **22.95** PPL (+4.8% vs F16)
|
||
|
||
The INT8 residual format (Q5_K_HIFI_RES8) efficiently preserves outlier precision with moderate overhead. However, imatrix quantization provides similar benefits through a different mechanism.
|
||
|
||
💡 **Key Finding:** For 0.6B models, **Q4_K_M + imatrix** is the recommended approach — it achieves slightly better quality than Q4_K_HIFI while being smaller and faster.
|
||
|
||
---
|
||
|
||
## Precision Loss Summary (vs F16 Baseline: PPL 21.8916)
|
||
|
||
| Model | PPL (no imatrix) | vs F16 | PPL (imatrix) | vs F16 |
|
||
|-----------|------------------|--------|---------------|--------|
|
||
| Q4_K_HIFI | 23.6556 | **+8.1%** | 22.9451 | **+4.8%** |
|
||
| Q4_K_M | 23.6856 | **+8.2%** | 22.9210 | **+4.7%** ✅ |
|
||
| Q4_K_S | 24.5475 | **+12.1%** | 23.3136 | **+6.5%** |
|
||
|
||
**Key Observations:**
|
||
- Without imatrix: All Q4_K variants lose 8-12% precision vs F16
|
||
- With imatrix: Precision loss drops to 4.7-6.5% — imatrix recovers ~3-5.5% of lost precision
|
||
- Q4_K_M (imatrix) achieves the **lowest precision loss** at only +4.7% vs F16
|
||
- Q4_K_S suffers the most, with +12.1% precision loss without imatrix
|
||
|
||
---
|
||
|
||
## Tensor Distribution
|
||
|
||
| Model | q4_K | q5_K | q6_K | Q5_K_HIFI_RES8 | f32 | Total |
|
||
|-----------|------|------|------|----------------|-----|-------|
|
||
| Q4_K_S | 190 | 7 | 1 | 0 | 113 | 311 |
|
||
| Q4_K_M | 169 | 0 | 29 | 0 | 113 | 311 |
|
||
| Q4_K_HIFI | 158 | 0 | 23 | 17 | 113 | 311 |
|
||
|
||
**Q4_K_HIFI Enhancement:** 17 critical tensors (output.weight, token_embd, attn_v layers) use Q5_K_HIFI_RES8 format with INT8 residuals + per-block scale for maximum precision.
|
||
|
||
---
|
||
|
||
## Addendum: Impact of imatrix on ALL Quantization Types
|
||
|
||
All three quantization types can now use imatrix. Here's how they compare:
|
||
|
||
### imatrix Perplexity Improvements
|
||
|
||
| Model | Without imatrix | With imatrix | Improvement | PPL vs F16 (imatrix) |
|
||
|-----------|-----------------|--------------|-------------|----------------------|
|
||
| Q4_K_HIFI | 23.6556 | **22.9451** | **-0.71 (3.0% better)** | **+4.8%** |
|
||
| Q4_K_M | 23.6856 | **22.9210** | **-0.76 (3.2% better)** | **+4.7%** ✅ |
|
||
| Q4_K_S | 24.5475 | **23.3136** | **-1.23 (5.0% better)** | **+6.5%** |
|
||
|
||
### Revised Comparison (All with imatrix)
|
||
|
||
| Model | PPL (imatrix) | vs Q4_K_HIFI | vs F16 | Size |
|
||
|-----------|---------------|--------------|--------|------|
|
||
| Q4_K_M | **22.9210** | **-0.024 (-0.1%)** ✅ | **+4.7%** ✅ | 456.11 MiB |
|
||
| Q4_K_HIFI | 22.9451 | baseline | +4.8% | 487.39 MiB |
|
||
| Q4_K_S | 23.3136 | +0.369 (+1.6%) | +6.5% | 443.30 MiB |
|
||
|
||
### Key Findings
|
||
|
||
**With imatrix, Q4_K_M beats Q4_K_HIFI:**
|
||
|
||
| Comparison | Without imatrix | With imatrix |
|
||
|------------|-----------------|--------------|
|
||
| Q4_K_HIFI vs Q4_K_M | **-0.1%** (essentially equal) | **+0.1%** (Q4_K_M wins!) |
|
||
| Q4_K_HIFI vs Q4_K_S | **-3.6%** (moderate advantage) | **-1.6%** (small advantage) |
|
||
|
||
### Revised Recommendations (When Using imatrix)
|
||
|
||
| Priority | Without imatrix | With imatrix |
|
||
|-------------------|-----------------|--------------|
|
||
| **Quality First** | Q4_K_M ≈ Q4_K_HIFI | **Q4_K_M** ✅ (best PPL!) |
|
||
| **Best Balance** | Q4_K_M | **Q4_K_M** ✅ |
|
||
| **Size/Speed** | Q4_K_S | Q4_K_S |
|
||
|
||
### Conclusion
|
||
|
||
**If you're using imatrix quantization:**
|
||
- **Q4_K_M (imatrix) is the winner** — 0.1% better quality than Q4_K_HIFI (imatrix)
|
||
- Q4_K_M is 6.9% smaller and 1.7% faster than Q4_K_HIFI
|
||
- The size/speed overhead of Q4_K_HIFI provides **no quality benefit** with imatrix
|
||
- **Q4_K_M + imatrix is the recommended choice for 0.6B models**
|
||
|
||
**If you're NOT using imatrix:**
|
||
- Q4_K_HIFI and Q4_K_M are essentially equal (0.1% difference)
|
||
- Q4_K_M is smaller and faster, making it the practical choice
|
||
- **Q4_K_M without imatrix is recommended** (simpler, smaller, faster, equal quality)
|
||
|
||
**Bottom line for 0.6B models:** Q4_K_M (with or without imatrix) is the best choice. Q4_K_HIFI's INT8 residual format doesn't provide meaningful benefits at this model scale.
|
||
|
||
---
|
||
|
||
## Appendix (Test Environment Details)
|
||
|
||
| Component | Specification |
|
||
|---------------|----------------------------------------|
|
||
| **OS** | Ubuntu 24.04.3 LTS |
|
||
| **CPU** | AMD EPYC 9254 24-Core Processor |
|
||
| **CPU Cores** | 96 cores (2 threads/core) |
|
||
| **RAM** | 1.0Ti |
|
||
| **GPU** | NVIDIA L40S × 2 |
|
||
| **VRAM** | 46068 MiB per GPU |
|
||
| **CUDA** | 12.9 |
|
||
| **Test Data** | wikitext-2-raw, 584 chunks |
|
||
| **Context** | 512 tokens |
|
||
| **Samples** | 200 per speed benchmark |
|
||
| **imatrix** | mixed-imatrix-dataset.txt, 4697 chunks |
|