30 lines
1.6 KiB
Markdown
30 lines
1.6 KiB
Markdown
---
|
|
base_model:
|
|
- prism-ml/Ternary-Bonsai-8B-gguf
|
|
pipeline_tag: text-generation
|
|
---
|
|
|
|
> [!NOTE]
|
|
> **Why this repo?**
|
|
> The original `prism-ml/Ternary-Bonsai-8B-gguf` requires a custom fork to run. This repository contains the modified GGUF files that work directly with the **upstream `llama.cpp`** out of the box using native ternary weights.
|
|
|
|
## 📦 Available Files
|
|
|
|
- **`Ternary-Bonsai-8B-TQ2_0`** (2.12 GB) - ⭐ **RECOMMENDED**
|
|
*Provides great results with an almost imperceptible difference in quality compared to the others. Best balance of size and performance.*
|
|
|
|
- **`Ternary-Bonsai-8B-TQ2_0-Q6out`** (2.47 GB)
|
|
*Slightly faster than the standard TQ2_0 (uses Q6 output tensors, speedup because the llama ternary kernels are terrible), but the difference is barely noticeable (~12%, 16 tps vs 18 tps on cpu).*
|
|
|
|
- **`Ternary-Bonsai-8B-Q4_0-lossless`** (4.61 GB)
|
|
*Practically meaningless for daily use. It's just the ternary format packed into standard 4-bit, taking up 4 GB of space without any real benefits.*
|
|
|
|
## 📊 Benchmarks
|
|
|
|
*Note: Im lazy so i haven't run a full benchmark suite on every model yet, so here are the only GPQA diamond results. If you run your own benchmarks, feel free to share them in the Community tab and I will add them!*
|
|
|
|
| Model | Size | GPQA Acc | Correct | Physics | Chemistry | Biology |
|
|
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
|
|
| **TQ2_0** | 2.12 GB | 34.34% | 68/198 | 32/86 | 26/93 | 10/19 |
|
|
| **TQ2_0-Q6out** | 2.47 GB | 34.34% | 68/198 | 32/86 | 26/93 | 10/19 |
|
|
| **Q4_0-lossless** | 4.61 GB | 34.34% | 68/198 | 36/86 | 23/93 | 9/19 | |