303 lines
15 KiB
Markdown
303 lines
15 KiB
Markdown
---
|
||
license: apache-2.0
|
||
tags:
|
||
- gguf
|
||
- qwen
|
||
- qwen3
|
||
- qwen3-0.6b
|
||
- qwen3-0.6b-gguf
|
||
- llama.cpp
|
||
- quantized
|
||
- text-generation
|
||
- chat
|
||
- edge-ai
|
||
- tiny-model
|
||
- imatrix
|
||
- Q3_HIFI
|
||
- Q4_HIFI
|
||
- Q5_HIFI
|
||
- outlier-aware
|
||
- high-fidelity
|
||
datasets:
|
||
- wikitext
|
||
- codeparrot
|
||
- openwebmath
|
||
quantization: Q3_HIFI
|
||
base_model: Qwen/Qwen3-0.6B
|
||
author: geoffmunn
|
||
pipeline_tag: text-generation
|
||
language:
|
||
- en
|
||
- zh
|
||
---
|
||
|
||
# Qwen3-0.6B-f16-GGUF
|
||
|
||
This is a **GGUF-quantized version** of the **[Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B)** language model — a compact **600-million-parameter** LLM designed for **ultra-fast inference on low-resource devices**.
|
||
|
||
Converted for use with `llama.cpp`, [LM Studio](https://lmstudio.ai), [OpenWebUI](https://openwebui.com), and [GPT4All](https://gpt4all.io), enabling private AI anywhere — even offline.
|
||
|
||
> ⚠️ **Note**: This is a *very small* model. It will not match larger models (e.g., 4B+) in reasoning, coding, or factual accuracy. However, it shines in **speed, portability, and efficiency**.
|
||
|
||
## Why Use a 0.6B Model?
|
||
|
||
While limited in capability compared to larger models, **Qwen3-0.6B** excels at:
|
||
- Running **instantly** on CPUs without GPU
|
||
- Fitting into **<2GB RAM**, even when quantized
|
||
- Enabling **offline AI on microcontrollers, phones, or edge devices**
|
||
- Serving as a **fast baseline** for lightweight NLP tasks (intent detection, short responses)
|
||
|
||
It’s ideal for:
|
||
- Chatbots with simple flows
|
||
- On-device assistants
|
||
- Educational demos
|
||
- Rapid prototyping
|
||
|
||
# HIFI Quantization: High-Fidelity Low-Bit Compression
|
||
|
||
This is a custom quantization type that was created specifically to test if it was possible to obtain higher precision than the standard options (Q3_K_M for example).
|
||
|
||
**HIFI** ("High-Fidelity") quantization intelligently preserves model quality during aggressive weight compression by applying **tiered precision allocation** to critical weights. Instead of uniform bit reduction across all parameters, HIFI:
|
||
|
||
1. **Identifies sensitivity**: Uses weight analysis (and optionally imatrix) to locate tensors most vulnerable to quantization error
|
||
2. **Applies residual correction**: For the most critical 2–6 tensors, stores a secondary 8-bit residual correction term (`*_HIFI_RES8` types) that recovers precision lost in the primary quantization pass
|
||
3. **Tiered allocation**: Combines base quantization (Q3_K/Q4_K/Q5_K) with elevated precision tensors (Q4_K/Q5_K/Q6_K) on sensitive layers
|
||
|
||
This approach delivers near-lossless quality at dramatically reduced memory footprints—typically **64–78% memory reduction** versus F16 with minimal quality degradation.
|
||
|
||
# Qwen3 0.6B Quantization Guide: Cross-Bit Summary & Recommendations
|
||
|
||
## Executive Summary
|
||
|
||
At 0.6B scale, **quantization sensitivity is extreme**—small models lose proportionally more precision than larger ones when compressed. **Q2_K is unusable at any variant** (+88–106% precision loss even with imatrix). Viable options start at Q3_K, with quality improving dramatically through Q4_K and Q5_K:
|
||
|
||
| Bit Width | Best Variant (+ imatrix) | Quality vs F16 | File Size | Speed | Memory | Viability |
|
||
|-----------|--------------------------|----------------|-----------|-------|--------|-----------|
|
||
| **Q5_K** | Q5_K_M + imatrix | **+2.74%** ✅ | 508 MiB | 603 TPS | 1,103 MiB | Excellent |
|
||
| **Q4_K** | Q4_K_M + imatrix | +4.82% ✅ | 456 MiB | 624 TPS | 1,038 MiB | Very Good |
|
||
| **Q3_K** | Q3_K_HIFI + imatrix | +6.4% ✅ | 442 MiB | **632 TPS** (fastest) | 1,167 MiB | Good |
|
||
| **Q2_K** | Q2_K_HIFI + imatrix | **+88.3%** ❌ | 364 MiB | 638 TPS | 946 MiB | **UNUSABLE** |
|
||
|
||
💡 **Critical insight**: Unlike larger models, **0.6B requires imatrix for Q3_K/Q4_K viability** (recovers 9–27% of lost precision). Q5_K variants remain usable without imatrix but still benefit measurably (+0.5–0.9% PPL improvement).
|
||
|
||
---
|
||
|
||
## Bit-Width Recommendations by Use Case
|
||
|
||
### ✅ Quality-Critical Applications
|
||
**→ Q5_K_M + imatrix**
|
||
- Only +2.74% precision loss vs F16 (PPL 22.49) — near-lossless for this scale
|
||
- 45% memory reduction (1,103 MiB vs 2,015 MiB)
|
||
- 51% faster than F16 (603 TPS)
|
||
- ⚠️ **Avoid Q5_K_HIFI** — provides *no meaningful advantage* over Q5_K_M (0.02% PPL difference within measurement noise) while requiring custom build and 3.8% larger size
|
||
|
||
### ⚖️ Best Overall Balance (Recommended Default)
|
||
**→ Q4_K_M + imatrix**
|
||
- Excellent +4.82% precision loss (PPL 22.95) — imperceptible degradation in practice
|
||
- Strong 624 TPS speed (+56% vs F16)
|
||
- Compact 456 MiB file size (67% smaller than F16)
|
||
- Standard llama.cpp compatibility — no custom builds needed
|
||
- Ideal for most development and production scenarios
|
||
|
||
### 🚀 Maximum Speed / Minimum Size
|
||
**→ Q3_K_HIFI + imatrix**
|
||
- **Unique win-win at 0.6B scale**: Fastest variant (632 TPS) **AND** best Q3 quality (+6.4% loss)
|
||
- Smallest viable footprint (442 MiB file, 1,167 MiB runtime)
|
||
- ⚠️ **Never use Q3_K_S without imatrix** — suffers catastrophic +63.1% quality loss (unusable)
|
||
|
||
### 📱 Extreme Memory Constraints (< 450 MiB)
|
||
**→ Q3_K_S + imatrix**
|
||
- Absolute smallest (366 MiB file, 1,095 MiB runtime)
|
||
- Acceptable +36.7% precision loss with imatrix (vs unusable +63.1% without)
|
||
- Only viable option under 450 MiB budget
|
||
|
||
### ⛔ Avoid Entirely
|
||
**→ All Q2_K variants**
|
||
- Minimum +88.3% precision loss even with imatrix (PPL 41.22)
|
||
- Output quality severely compromised — incoherent generations expected
|
||
- **Minimum viable quantization for 0.6B is Q3_K_S with imatrix**
|
||
|
||
---
|
||
|
||
## Critical Warnings for 0.6B Scale
|
||
|
||
⚠️ **Q2_K is unusable at 0.6B scale** — Do not deploy under any circumstances. Even Q2_K_HIFI + imatrix suffers +88.3% precision loss. This is a hard floor — Q3_K is the minimum viable quantization level.
|
||
|
||
⚠️ **imatrix is non-optional for Q3_K/Q4_K** — Without it:
|
||
- Q3_K variants lose 15.9–63.1% precision (borderline unusable)
|
||
- Q4_K variants lose 8.1–12.2% precision (significant degradation)
|
||
- *All recover 9–27% of lost precision with imatrix at zero inference cost*
|
||
|
||
⚠️ **HIFI variants provide negligible benefit at 0.6B**:
|
||
- Q5_K_HIFI differs from Q5_K_M by only **1 tensor** (168 vs 169 q5_K)
|
||
- Q4_K_HIFI differs from Q4_K_M by marginal tensor allocation changes
|
||
- Quality differences are **within measurement noise** (±0.20 PPL)
|
||
- Costs 3.8–6.9% more size and requires custom build — **not worth it**
|
||
|
||
⚠️ **Q3_K_HIFI uniquely breaks the quality/speed tradeoff** at 0.6B:
|
||
- Unlike larger models where HIFI is slower, at 0.6B it's the *fastest* Q3 variant (+2.4% vs Q3_K_M)
|
||
- This anomaly occurs because tensor allocation differences compress to minimal overhead at tiny scales
|
||
|
||
⚠️ **Small models ≠ large models** — Quantization behavior differs fundamentally:
|
||
- At 0.6B: HIFI variants provide negligible benefit; Q2_K is unusable
|
||
- At 8B+: HIFI variants deliver measurable quality gains; Q2_K may be viable for specific tasks
|
||
- Never assume quantization patterns scale linearly across model sizes
|
||
|
||
---
|
||
|
||
## Memory Budget Guide
|
||
|
||
| Available VRAM | Recommended Variant | Expected Quality | Why |
|
||
|----------------|---------------------|------------------|-----|
|
||
| **< 450 MiB** | Q3_K_S + imatrix | PPL 29.92, +36.7% loss ⚠️ | Only option that fits; quality acceptable for non-critical tasks |
|
||
| **450 – 600 MiB** | Q3_K_HIFI + imatrix | PPL 23.29, +6.4% loss ✅ | Best Q3 quality; unique speed/quality win-win |
|
||
| **600 – 800 MiB** | Q4_K_M + imatrix | PPL 22.95, +4.82% loss ✅ | Best balance of quality/speed/size; standard compatibility |
|
||
| **800 – 1,200 MiB** | Q5_K_M + imatrix | PPL 22.49, +2.74% loss ✅ | Near-lossless quality; best precision available |
|
||
| **> 1,200 MiB** | F16 | PPL 21.89, 0% loss | F16 is only 1.4 GiB total — viable baseline for tiny models |
|
||
|
||
---
|
||
|
||
## Decision Flowchart
|
||
|
||
```mermaid
|
||
Memory < 450 MiB?
|
||
├─ Yes → Q3_K_S + imatrix (only option; +36.7% loss)
|
||
└─ No → Need best quality?
|
||
├─ Yes → Q5_K_M + imatrix (+2.74% loss)
|
||
└─ No → Need max speed?
|
||
├─ Yes → Q3_K_HIFI + imatrix (632 TPS, +6.4% loss)
|
||
└─ No → Q4_K_M + imatrix (best balance, +4.82% loss)
|
||
```
|
||
|
||
---
|
||
|
||
## Cross-Bit Performance Comparison
|
||
|
||
| Priority | Q3_K Best | Q4_K Best | Q5_K Best | Winner |
|
||
|----------|-----------|-----------|-----------|--------|
|
||
| **Quality (with imat)** | Q3_K_HIFI (+6.4%) | Q4_K_M (+4.82%) | **Q5_K_M (+2.74%)** ✅ | **Q5_K_M** |
|
||
| **Speed** | **Q3_K_HIFI (632 TPS)** ✅ | Q4_K_S (624 TPS) | Q5_K_S (607 TPS) | **Q3_K_HIFI** |
|
||
| **Smallest Size** | **Q3_K_S (366 MiB)** ✅ | Q4_K_S (416 MiB) | Q5_K_S (501 MiB) | **Q3_K_S** |
|
||
| **Best Balance** | Q3_K_HIFI + imat | **Q4_K_M + imat** ✅ | Q5_K_M + imat | **Q4_K_M** |
|
||
| **Viability Floor** | Q3_K_S + imat ✅ | Q4_K_S + imat ✅ | Q5_K_S ✅ | **Q3 minimum** |
|
||
|
||
✅ = Recommended for general use
|
||
⚠️ = Context-dependent (see warnings above)
|
||
❌ = Unusable (Q2_K variants)
|
||
|
||
---
|
||
|
||
## Bottom Line Recommendations
|
||
|
||
| Scenario | Recommended Variant | Rationale |
|
||
|----------|---------------------|-----------|
|
||
| **Default / General Purpose** | Q4_K_M + imatrix | Best balance of quality (+4.82%), speed (624 TPS), size (456 MiB), and compatibility |
|
||
| **Maximum Quality** | Q5_K_M + imatrix | Near-lossless (+2.74% vs F16) with modest size/speed trade-offs |
|
||
| **Maximum Speed** | Q3_K_HIFI + imatrix | Fastest (632 TPS) with surprisingly good quality (+6.4% loss) |
|
||
| **Minimum Size** | Q3_K_S + imatrix | Smallest footprint (366 MiB) — only if memory < 450 MiB |
|
||
| **Avoid Entirely** | All Q2_K variants | Unusable quality even with imatrix (+88%+ loss) |
|
||
|
||
⚠️ **Golden rules for 0.6B**:
|
||
1. **Never use Q2_K** — minimum viable quantization is Q3_K_S *with imatrix*
|
||
2. **Always use imatrix with Q3_K/Q4_K** — quality degradation without it is severe and avoidable
|
||
3. **Skip HIFI variants** — provide negligible benefit at this scale while requiring custom builds
|
||
4. **F16 is viable** — at only 1.4 GiB total, consider F16 as baseline if quality is paramount
|
||
|
||
✅ **0.6B quantization reality check**: This scale is highly sensitive to compression. While Q5_K_M + imatrix delivers excellent results (+2.74% loss), the absolute quality floor is much higher than at larger scales. For production work requiring reliable output, **Q4_K_M + imatrix is the pragmatic sweet spot** — excellent quality with robust compatibility and minimal constraints.
|
||
|
||
## Non-technical model anaysis and rankings
|
||
|
||
**NOTE:** This analysis does not include the HIFI models.
|
||
|
||
I have run each of these models across 6 questions, and ranked them all based on the quality of the anwsers.
|
||
**Qwen3-0.6B-f16:Q5_K_M** is the best model across all question types, but if you want to play it safe with a higher precision model, then you could consider using **Qwen3-0.6B-f16:Q8_0**.
|
||
|
||
You can read the results here: [Qwen3-0.6b-f16-analysis.md](Qwen3-0.6b-f16-analysis.md)
|
||
|
||
If you find this useful, please give the project a ❤️ like.
|
||
|
||
## Non-HIFI recommentation table based on output
|
||
|
||
| Level | Speed | Size | Recommendation |
|
||
|-----------|-----------|------------|--------------------------------------------------------------------|
|
||
| Q2_K | ⚡ Fastest | 347 MB | 🚨 **DO NOT USE.** Could not provide an answer to any question. |
|
||
| Q3_K_S | ⚡ Fast | 390 MB | Not recommended, did not appear in any top 3 results. |
|
||
| Q3_K_M | ⚡ Fast | 414 MB | First place in the bat & ball question, no other top 3 appearances.|
|
||
| Q4_K_S | 🚀 Fast | 471 MB | A good option for technical, low-temperature questions. |
|
||
| Q4_K_M | 🚀 Fast | 484 MB | Showed up in a few results, but not recommended. |
|
||
| 🥈 Q5_K_S | 🐢 Medium | 544 MB | 🥈 A very close second place. Good for all query types. |
|
||
| 🥇 Q5_K_M | 🐢 Medium | 551 MB | 🥇 **Best overall model.** Highly recommended for all query types. |
|
||
| Q6_K | 🐌 Slow | 623 MB | Showed up in a few results, but not recommended. |
|
||
| 🥉 Q8_0 | 🐌 Slow | 805 MB | 🥉 Very good for non-technical, creative-style questions. |
|
||
|
||
## Build notes
|
||
|
||
You can read the guide for building llama.cpp here: [HIFI_BUILD_GUIDE.md](https://github.com/geoffmunn/llama.cpp/blob/master/HIFI_BUILD_GUIDE.md).
|
||
|
||
The HIFI quantization also used a massive 9343 chunk imatrix file for extra precision. You can re-use it here: [Qwen3-0.6B-f16-imatrix-9343-generic.gguf](https://huggingface.co/geoffmunn/Qwen3-0.6B-f16/blob/main/Qwen3-0.6B-f16-imatrix-9343-generic.gguf)
|
||
|
||
The imatrix was created as a generic mix of Wikipedia, mathmatics, and coding examples.
|
||
|
||
### Source code
|
||
|
||
You can use the HIFI GitHub repository to build it from source if you're interested: [https://github.com/geoffmunn/llama.cpp](https://github.com/geoffmunn/llama.cpp).
|
||
|
||
Build notes: [HIFI_BUILD_GUIDE.md](https://github.com/geoffmunn/llama.cpp/blob/master/HIFI_BUILD_GUIDE.md)
|
||
|
||
Improvements and feedback are welcome.
|
||
|
||
## Usage
|
||
|
||
Load this model using:
|
||
- [OpenWebUI](https://openwebui.com) – self-hosted AI interface with RAG & tools
|
||
- [LM Studio](https://lmstudio.ai) – desktop app with GPU support and chat templates
|
||
- [GPT4All](https://gpt4all.io) – private, local AI chatbot (offline-first)
|
||
- Or directly via `llama.cpp`
|
||
|
||
Each quantized model includes its own `README.md` and shares a common `MODELFILE` for optimal configuration.
|
||
|
||
Importing directly into Ollama should work, but you might encounter this error: `Error: invalid character '<' looking for beginning of value`.
|
||
In this case try these steps:
|
||
|
||
1. `wget https://huggingface.co/geoffmunn/Qwen3-0.6B-f16/resolve/main/Qwen3-0.6B-f16%3AQ3_K_M.gguf` (replace the quantised version with the one you want)
|
||
2. `nano Modelfile` and enter these details (again, replacing Q3_K_M with the version you want):
|
||
```text
|
||
FROM ./Qwen3-0.6B-f16:Q3_K_M.gguf
|
||
|
||
# Chat template using ChatML (used by Qwen)
|
||
SYSTEM You are a helpful assistant
|
||
|
||
TEMPLATE "{{ if .System }}<|im_start|>system
|
||
{{ .System }}<|im_end|>{{ end }}<|im_start|>user
|
||
{{ .Prompt }}<|im_end|>
|
||
<|im_start|>assistant
|
||
"
|
||
PARAMETER stop <|im_start|>
|
||
PARAMETER stop <|im_end|>
|
||
|
||
# Default sampling
|
||
PARAMETER temperature 0.6
|
||
PARAMETER top_p 0.95
|
||
PARAMETER top_k 20
|
||
PARAMETER min_p 0.0
|
||
PARAMETER repeat_penalty 1.1
|
||
PARAMETER num_ctx 4096
|
||
```
|
||
|
||
The `num_ctx` value has been dropped to increase speed significantly.
|
||
|
||
3. Then run this command: `ollama create Qwen3-0.6B-f16:Q3_K_M -f Modelfile`
|
||
|
||
You will now see "Qwen3-0.6B-f16:Q3_K_M" in your Ollama model list.
|
||
|
||
These import steps are also useful if you want to customise the default parameters or system prompt.
|
||
|
||
## Author
|
||
|
||
👤 Geoff Munn (@geoffmunn)
|
||
🔗 [Hugging Face Profile](https://huggingface.co/geoffmunn)
|
||
|
||
## Disclaimer
|
||
|
||
This is a community conversion for local inference. Not affiliated with Alibaba Cloud or the Qwen team.
|