80 lines
3.4 KiB
Markdown
80 lines
3.4 KiB
Markdown
|
---
|
|||
|
|
language: [en, zh]
|
|||
|
|
license: apache-2.0
|
|||
|
|
library_name: gguf
|
|||
|
|
base_model: Qwen/Qwen3-4B-Instruct-2507
|
|||
|
|
tags: [quantized, gguf, mixed-precision, bitclass, qwen3]
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# Qwen3-4B-Instruct-2507 — BitClass2 Mixed-Precision GGUF
|
|||
|
|
|
|||
|
|
Mixed-precision GGUF quantizations of [Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507)
|
|||
|
|
using **Hessian-informed per-tensor bit allocation**. Each tensor group receives
|
|||
|
|
the precision level that minimizes quality loss for its measured sensitivity —
|
|||
|
|
more bits where they matter, fewer where they don't.
|
|||
|
|
|
|||
|
|
## Available Quantizations
|
|||
|
|
|
|||
|
|
| File | BPW | Size | PPL ↓ | tok/s | Use Case |
|
|||
|
|
|---|---|---|---|---|---|
|
|||
|
|
| [`Qwen3-4B-Instruct-2507-Q8_0.gguf`](./Qwen3-4B-Instruct-2507-Q8_0.gguf) | 8.5 | 4.28 GB | 2.651 | 11.4 | Near-lossless reference |
|
|||
|
|
| [`Qwen3-4B-Instruct-2507-Q6_K.gguf`](./Qwen3-4B-Instruct-2507-Q6_K.gguf) | 5.8 | 2.93 GB | 2.888 | 13.6 | High quality, moderate size |
|
|||
|
|
| [`Qwen3-4B-Instruct-2507-Q5_K_M.gguf`](./Qwen3-4B-Instruct-2507-Q5_K_M.gguf) | 5.2 | 2.60 GB | 2.971 | 14.3 | Balanced quality and size |
|
|||
|
|
| [`Qwen3-4B-Instruct-2507-Q4_K_M.gguf`](./Qwen3-4B-Instruct-2507-Q4_K_M.gguf) | 4.7 | 2.35 GB | 2.978 | 14.1 | Best quality-to-size ratio |
|
|||
|
|
| [`Qwen3-4B-Instruct-2507-Q3_K_S.gguf`](./Qwen3-4B-Instruct-2507-Q3_K_S.gguf) | 3.2 | 1.62 GB | 3.214 | 18.9 | Maximum compression |
|
|||
|
|
|
|||
|
|
**Recommended:** Q4_K_M for the best quality-to-size ratio (PPL 2.978 at just 2.35 GB).
|
|||
|
|
Q3_K_S for maximum compression. Q6_K for high quality.
|
|||
|
|
|
|||
|
|
## How It Works
|
|||
|
|
|
|||
|
|
Standard quantization applies one precision level uniformly across all tensors.
|
|||
|
|
BitClass2 uses **Hessian-based sensitivity analysis** (H_diag = mean(X²) per layer)
|
|||
|
|
to identify which tensors lose the most quality when quantized, then solves an
|
|||
|
|
LP-optimal knapsack allocation: minimize Σ(sensitivity × quantization_error)
|
|||
|
|
subject to total size ≤ target. Sensitive tensors get higher precision,
|
|||
|
|
insensitive ones get lower precision, at the same total file size.
|
|||
|
|
|
|||
|
|
Within each suffix group, the fractional BPW planner further varies types
|
|||
|
|
per-layer using blended imatrix + Hessian scores, so late attention layers
|
|||
|
|
(most sensitive) get higher precision than middle layers (least sensitive).
|
|||
|
|
|
|||
|
|
## Key Sensitivity Findings (Qwen3-4B)
|
|||
|
|
|
|||
|
|
- **Late attention layers (29-35) are most sensitive** — blk.34 k/v score 1.0
|
|||
|
|
- **down_proj is the most sensitive MLP tensor** — projects back to residual stream
|
|||
|
|
- **gate_proj/up_proj are least sensitive** — safe to quantize aggressively
|
|||
|
|
- **K > V for attention weight sensitivity** — k_proj averages 0.66 vs v_proj 0.50
|
|||
|
|
|
|||
|
|
## Usage
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
# Download
|
|||
|
|
huggingface-cli download sh111111111111111/Qwen3-4B-Instruct-2507-BitClass2-GGUF \
|
|||
|
|
Qwen3-4B-Instruct-2507-Q4_K_M.gguf --local-dir .
|
|||
|
|
|
|||
|
|
# Chat with llama.cpp
|
|||
|
|
llama-cli -m Qwen3-4B-Instruct-2507-Q4_K_M.gguf -cnv
|
|||
|
|
|
|||
|
|
# Serve via API
|
|||
|
|
llama-server -m Qwen3-4B-Instruct-2507-Q4_K_M.gguf --port 8080
|
|||
|
|
|
|||
|
|
# Ollama
|
|||
|
|
ollama run hf.co/sh111111111111111/Qwen3-4B-Instruct-2507-BitClass2-GGUF:Qwen3-4B-Instruct-2507-Q4_K_M.gguf
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## Benchmark Details
|
|||
|
|
|
|||
|
|
All benchmarks run on NVIDIA GB10 ATOM (128GB unified memory, aarch64).
|
|||
|
|
llama.cpp commit 406f4e3. PPL via `llama-perplexity` (2 chunks, 851 context).
|
|||
|
|
tok/s via `llama-bench` (tg128, ngl=999).
|
|||
|
|
|
|||
|
|
## Disclaimer
|
|||
|
|
|
|||
|
|
Independent project. Not affiliated with or endorsed by Qwen, Unsloth, ByteShape, Bartowski, or llama.cpp.
|
|||
|
|
|
|||
|
|
## License
|
|||
|
|
|
|||
|
|
Apache 2.0, inherited from [Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507).
|