49 lines
1.5 KiB
Markdown
49 lines
1.5 KiB
Markdown
---
|
|
license: apache-2.0
|
|
base_model: Kolyadual/MIXdevAI-yandexGPT5-8B
|
|
tags:
|
|
- gguf
|
|
- llama.cpp
|
|
- text-generation
|
|
---
|
|
|
|
# MIXdevAI-yandexGPT5-8B-GGUF
|
|
|
|
Full set of GGUF quantizations for [Kolyadual/MIXdevAI-yandexGPT5-8B](https://huggingface.co/Kolyadual/MIXdevAI-yandexGPT5-8B).
|
|
|
|
## Files
|
|
|
|
| File | Quantization | Type |
|
|
|---|---|---|
|
|
| `mixdev-ya-f16.gguf` | F16 | Base Model |
|
|
| `mixdev-ya-q8_0.gguf` | Q8_0 | Text Model |
|
|
| `mixdev-ya-q6_k.gguf` | Q6_K | Text Model |
|
|
| `mixdev-ya-q5_k_m.gguf` | Q5_K_M | Text Model |
|
|
| `mixdev-ya-q5_k_s.gguf` | Q5_K_S | Text Model |
|
|
| `mixdev-ya-q5_1.gguf` | Q5_1 | Text Model |
|
|
| `mixdev-ya-q5_0.gguf` | Q5_0 | Text Model |
|
|
| `mixdev-ya-q4_k_m.gguf` | Q4_K_M | Text Model |
|
|
| `mixdev-ya-q4_k_s.gguf` | Q4_K_S | Text Model |
|
|
| `mixdev-ya-q4_1.gguf` | Q4_1 | Text Model |
|
|
| `mixdev-ya-q4_0.gguf` | Q4_0 | Text Model |
|
|
| `mixdev-ya-q3_k_l.gguf` | Q3_K_L | Text Model |
|
|
| `mixdev-ya-q3_k_m.gguf` | Q3_K_M | Text Model |
|
|
| `mixdev-ya-q3_k_s.gguf` | Q3_K_S | Text Model |
|
|
| `mixdev-ya-q2_k.gguf` | Q2_K | Text Model |
|
|
|
|
### Quantization notes
|
|
- **Q8_0 / F16**: Almost lossless. Best quality, largest size.
|
|
- **Q6_K / Q5_K_M**: Excellent balance between quality and size.
|
|
- **Q4_K_M**: Golden standard for local inference.
|
|
- **Q3_K_M / Q2_K**: Noticeable quality degradation. Use only if you have severe RAM/VRAM constraints.
|
|
|
|
## How to run locally
|
|
|
|
Example using `llama.cpp` server:
|
|
|
|
```bash
|
|
llama-server \
|
|
-m mixdev-ya-q4_k_m.gguf \
|
|
-ngl 999 \
|
|
--host 0.0.0.0 --port 8080 \
|
|
-c 32768 |