79 lines
3.5 KiB
Markdown
79 lines
3.5 KiB
Markdown
|
---
|
|||
|
|
language:
|
|||
|
|
- en
|
|||
|
|
library_name: gguf
|
|||
|
|
tags:
|
|||
|
|
- MoQ
|
|||
|
|
- mixture-of-quants
|
|||
|
|
- GGUF
|
|||
|
|
- QWEN
|
|||
|
|
- quantization
|
|||
|
|
base_model:
|
|||
|
|
- LiquidAI/LFM2.5-8B-A1B
|
|||
|
|
license: mit
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# 🚀 MoQ: Mixture of Quants
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
>MoQ (Mixture of Quants) is a smart way to shrink AI models without losing their "brainpower." Unlike old methods that treat every part of the model the same, MoQ identifies the most important parts and keeps them high-quality, while heavily compressing the rest to save space.
|
|||
|
|
|
|||
|
|
The result? A model that punches significantly above its weight class.
|
|||
|
|
|
|||
|
|
Benjamin Marie evaluated MoQ GGUFs ("Mixture of Quants") against Unsloth Dynamic (UD) quants, focusing on low-bit versions below 4 bits on average — the range where GGUF models typically struggle most.
|
|||
|
|
Results: At similar bits-per-weight (Bpw), MoQ outperforms Unsloth Dynamic quants by ~10% on benchmarks, while also being roughly 2× more token-efficient on average.
|
|||
|
|
|
|||
|
|
"MoQ models are much better than UD quants on benchmarks, and they are also more token-efficient."
|
|||
|
|
## Comparison
|
|||
|
|
Here is the comparison between MoQ and Unsloth dynamic quants for LFM 2.5 8BA1B.
|
|||
|
|
MoQ perform better i guess .
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
Thanks to benjamin Marie for his evals
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
| Folder Link | BPW | Total Size | * |
|
|||
|
|
| :--- | :---: | :---: | :--- |
|
|||
|
|
| [📂 **Root**](https://huggingface.co/w-ahmad/LFM2.5-8B-A1B-GGUF-MoQ/blob/main/MoQ-2.75.gguf) | **2.75** | **2.91 GB**
|
|||
|
|
| [📂 **Root**](https://huggingface.co/w-ahmad/LFM2.5-8B-A1B-GGUF-MoQ/blob/main/MoQ-3.0.gguf) | **3.0** | **3.17 GB**
|
|||
|
|
| [📂 **Root**](https://huggingface.co/w-ahmad/LFM2.5-8B-A1B-GGUF-MoQ/blob/main/MoQ-3.25.gguf) | **3.25** | **3.43 GB**
|
|||
|
|
| [📂 **Root**](https://huggingface.co/w-ahmad/LFM2.5-8B-A1B-GGUF-MoQ/blob/main/MoQ-3.5.gguf) | **3.5** | **3.71 GB**
|
|||
|
|
| [📂 **Root**](https://huggingface.co/w-ahmad/LFM2.5-8B-A1B-GGUF-MoQ/blob/main/MoQ-3.75.gguf) | **3.75** | **3.96 GB**
|
|||
|
|
| [📂 **Root**](https://huggingface.co/w-ahmad/LFM2.5-8B-A1B-GGUF-MoQ/blob/main/MoQ-4.0.gguf) | **4.0** | **4.20 GB**
|
|||
|
|
| [📂 **Root**](https://huggingface.co/w-ahmad/LFM2.5-8B-A1B-GGUF-MoQ/blob/main/MoQ-4.25.gguf) | **4.25** | **4.46 GB**
|
|||
|
|
| [📂 **Root**](https://huggingface.co/w-ahmad/LFM2.5-8B-A1B-GGUF-MoQ/blob/main/MoQ-4.5.gguf) | **4.5** | **4.72 GB**
|
|||
|
|
| [📂 **Root**](https://huggingface.co/w-ahmad/LFM2.5-8B-A1B-GGUF-MoQ/blob/main/MoQ-5.0.gguf) | **5.0** | **5.30 GB**
|
|||
|
|
| [📂 **bf16**](https://huggingface.co/w-ahmad/LFM2.5-8B-A1B-GGUF-MoQ/blob/main/bf16/LFM2.5-8B-A1B-BF16.gguf) | **16.0** | **16.95 GB**
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
## 🧠 The MoQ Edge
|
|||
|
|
MoQ optimizes the architecture for the **Pareto frontier** of memory and performance.
|
|||
|
|
|
|||
|
|
* **Dynamic Bitrate Allocation:** No more "one-size-fits-all." MoQ assigns precision where it actually matters.
|
|||
|
|
* **Cognitive Preservation:** Massive VRAM savings with near-zero degradation in logic and coherence.
|
|||
|
|
* **Next-Gen Efficiency:** Fits "Large" model intelligence into "Small" model hardware.
|
|||
|
|
|
|||
|
|
|
|||
|
|
##
|
|||
|
|
x : https://x.com/WaleedAhmad1a10
|
|||
|
|
If MoQ does not perform well, email me :
|
|||
|
|
waleedahmad.1a10@gmail.com
|
|||
|
|
|
|||
|
|
## 🛠 Usage & Deployment.
|
|||
|
|
|
|||
|
|
```bash.
|
|||
|
|
./llama-cli -m Qwen3.5-9B-MoQ-4.0.gguf -p "The future of efficient AI is..."
|