初始化项目,由ModelHub XC社区提供模型
Model: kirilldual0987/MIXdevAI-yandexGPT5-8B-GGUF Source: Original Platform
This commit is contained in:
49
README.md
Normal file
49
README.md
Normal file
@@ -0,0 +1,49 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
base_model: Kolyadual/MIXdevAI-yandexGPT5-8B
|
||||
tags:
|
||||
- gguf
|
||||
- llama.cpp
|
||||
- text-generation
|
||||
---
|
||||
|
||||
# MIXdevAI-yandexGPT5-8B-GGUF
|
||||
|
||||
Full set of GGUF quantizations for [Kolyadual/MIXdevAI-yandexGPT5-8B](https://huggingface.co/Kolyadual/MIXdevAI-yandexGPT5-8B).
|
||||
|
||||
## Files
|
||||
|
||||
| File | Quantization | Type |
|
||||
|---|---|---|
|
||||
| `mixdev-ya-f16.gguf` | F16 | Base Model |
|
||||
| `mixdev-ya-q8_0.gguf` | Q8_0 | Text Model |
|
||||
| `mixdev-ya-q6_k.gguf` | Q6_K | Text Model |
|
||||
| `mixdev-ya-q5_k_m.gguf` | Q5_K_M | Text Model |
|
||||
| `mixdev-ya-q5_k_s.gguf` | Q5_K_S | Text Model |
|
||||
| `mixdev-ya-q5_1.gguf` | Q5_1 | Text Model |
|
||||
| `mixdev-ya-q5_0.gguf` | Q5_0 | Text Model |
|
||||
| `mixdev-ya-q4_k_m.gguf` | Q4_K_M | Text Model |
|
||||
| `mixdev-ya-q4_k_s.gguf` | Q4_K_S | Text Model |
|
||||
| `mixdev-ya-q4_1.gguf` | Q4_1 | Text Model |
|
||||
| `mixdev-ya-q4_0.gguf` | Q4_0 | Text Model |
|
||||
| `mixdev-ya-q3_k_l.gguf` | Q3_K_L | Text Model |
|
||||
| `mixdev-ya-q3_k_m.gguf` | Q3_K_M | Text Model |
|
||||
| `mixdev-ya-q3_k_s.gguf` | Q3_K_S | Text Model |
|
||||
| `mixdev-ya-q2_k.gguf` | Q2_K | Text Model |
|
||||
|
||||
### Quantization notes
|
||||
- **Q8_0 / F16**: Almost lossless. Best quality, largest size.
|
||||
- **Q6_K / Q5_K_M**: Excellent balance between quality and size.
|
||||
- **Q4_K_M**: Golden standard for local inference.
|
||||
- **Q3_K_M / Q2_K**: Noticeable quality degradation. Use only if you have severe RAM/VRAM constraints.
|
||||
|
||||
## How to run locally
|
||||
|
||||
Example using `llama.cpp` server:
|
||||
|
||||
```bash
|
||||
llama-server \
|
||||
-m mixdev-ya-q4_k_m.gguf \
|
||||
-ngl 999 \
|
||||
--host 0.0.0.0 --port 8080 \
|
||||
-c 32768
|
||||
Reference in New Issue
Block a user