commit e235d06b36f38e6a654ee34f26597db8038e10c7 Author: ModelHub XC Date: Sat Aug 8 00:39:19 2026 +0800 初始化项目，由ModelHub XC社区提供模型 Model: DhruvalLabs/qwen3-8b-claude-agentic-fable5-GGUF Source: Original Platform diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..a188b34 --- /dev/null +++ b/.gitattributes @@ -0,0 +1,45 @@ +*.7z filter=lfs diff=lfs merge=lfs -text +*.arrow filter=lfs diff=lfs merge=lfs -text +*.bin filter=lfs diff=lfs merge=lfs -text +*.bz2 filter=lfs diff=lfs merge=lfs -text +*.ckpt filter=lfs diff=lfs merge=lfs -text +*.ftz filter=lfs diff=lfs merge=lfs -text +*.gz filter=lfs diff=lfs merge=lfs -text +*.h5 filter=lfs diff=lfs merge=lfs -text +*.joblib filter=lfs diff=lfs merge=lfs -text +*.lfs.* filter=lfs diff=lfs merge=lfs -text +*.mlmodel filter=lfs diff=lfs merge=lfs -text +*.model filter=lfs diff=lfs merge=lfs -text +*.msgpack filter=lfs diff=lfs merge=lfs -text +*.npy filter=lfs diff=lfs merge=lfs -text +*.npz filter=lfs diff=lfs merge=lfs -text +*.onnx filter=lfs diff=lfs merge=lfs -text +*.ot filter=lfs diff=lfs merge=lfs -text +*.parquet filter=lfs diff=lfs merge=lfs -text +*.pb filter=lfs diff=lfs merge=lfs -text +*.pickle filter=lfs diff=lfs merge=lfs -text +*.pkl filter=lfs diff=lfs merge=lfs -text +*.pt filter=lfs diff=lfs merge=lfs -text +*.pth filter=lfs diff=lfs merge=lfs -text +*.rar filter=lfs diff=lfs merge=lfs -text +*.safetensors filter=lfs diff=lfs merge=lfs -text +saved_model/**/* filter=lfs diff=lfs merge=lfs -text +*.tar.* filter=lfs diff=lfs merge=lfs -text +*.tar filter=lfs diff=lfs merge=lfs -text +*.tflite filter=lfs diff=lfs merge=lfs -text +*.tgz filter=lfs diff=lfs merge=lfs -text +*.wasm filter=lfs diff=lfs merge=lfs -text +*.xz filter=lfs diff=lfs merge=lfs -text +*.zip filter=lfs diff=lfs merge=lfs -text +*.zst filter=lfs diff=lfs merge=lfs -text +*tfevents* filter=lfs diff=lfs merge=lfs -text +qwen3-8b-claude-agentic-fable5-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text +qwen3-8b-claude-agentic-fable5-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text +qwen3-8b-claude-agentic-fable5-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text +qwen3-8b-claude-agentic-fable5-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text +qwen3-8b-claude-agentic-fable5-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text +qwen3-8b-claude-agentic-fable5-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text +qwen3-8b-claude-agentic-fable5-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text +qwen3-8b-claude-agentic-fable5-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text +qwen3-8b-claude-agentic-fable5-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text +qwen3-8b-claude-agentic-fable5-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text diff --git a/README.md b/README.md new file mode 100644 index 0000000..6878955 --- /dev/null +++ b/README.md @@ -0,0 +1,165 @@ +--- +license: apache-2.0 +base_model: DhruvalLabs/qwen3-8b-claude-agentic-fable5 +pipeline_tag: text-generation +tags: + - base_model:adapter:Qwen/Qwen3-8B + - text-generation-inference + - quantized + - qwen3 + - safetensors + - gguf + - conversational + - region:us + - unsloth + - tool-use + - transformers + - lora + - finetuned + - claude + - agentic + - license:apache-2.0 + - en + - coding-agent + - base_model:Qwen/Qwen3-8B + - text-generation + - fable5 +language: + - en +--- + +
+ +# qwen3-8b-claude-agentic-fable5 — GGUF Quantizations + +[![Model on HF](https://img.shields.io/badge/🤗-Model_on_HuggingFace-yellow)](https://huggingface.co/DhruvalLabs/qwen3-8b-claude-agentic-fable5-GGUF) +[![Original Model](https://img.shields.io/badge/Original-DhruvalLabs_qwen3-8b-claude-agentic-fable5-blue)](https://huggingface.co/DhruvalLabs/qwen3-8b-claude-agentic-fable5) +[![quant-kit](https://img.shields.io/badge/Made_with-quant--kit-green)](https://github.com/DhruvalPtl/quant-kit) + +**Quantized GGUF versions of [DhruvalLabs/qwen3-8b-claude-agentic-fable5](https://huggingface.co/DhruvalLabs/qwen3-8b-claude-agentic-fable5)** + +Works with **[llama.cpp](https://github.com/ggerganov/llama.cpp)** · **[Ollama](https://ollama.ai)** · **[LM Studio](https://lmstudio.ai)** · **[Open WebUI](https://openwebui.com)** · **[Jan](https://jan.ai)** + +*Quantized by **[DhruvalLabs](https://huggingface.co/DhruvalLabs)** on July 07, 2026 using [quant-kit](https://github.com/DhruvalPtl/quant-kit)* + +
+ +--- + +## ⚖️ The Pareto Frontier — Efficiency vs Intelligence + +> Can you run a powerful model on a laptop without losing its intelligence? + +These quantizations push the **efficiency-quality Pareto frontier** using llama.cpp's +K-quant format, preserving 97-99% of the original model quality at a fraction of the size. + +| Benchmark | Original (FP16) | Q4_K_M | Quality Retained | +|---|---|---|---| +| **MMLU Pro** | *See [original card](https://huggingface.co/DhruvalLabs/qwen3-8b-claude-agentic-fable5)* | *Run benchmarks* | ~97-99% | +| **HellaSwag** | *See [original card](https://huggingface.co/DhruvalLabs/qwen3-8b-claude-agentic-fable5)* | *Run benchmarks* | ~97-99% | +| **ARC Challenge** | *See [original card](https://huggingface.co/DhruvalLabs/qwen3-8b-claude-agentic-fable5)* | *Run benchmarks* | ~97-99% | +| **TruthfulQA** | *See [original card](https://huggingface.co/DhruvalLabs/qwen3-8b-claude-agentic-fable5)* | *Run benchmarks* | ~97-99% | +| **GSM8K** | *See [original card](https://huggingface.co/DhruvalLabs/qwen3-8b-claude-agentic-fable5)* | *Run benchmarks* | ~97-99% | + + +--- + +## 📦 Available Files + +| Filename | Size | RAM Required | Quant | Quality | Best For | +|---|---|---|---|---|---| +| `qwen3-8b-claude-agentic-fable5-Q2_K.gguf` | 3.06 GB | ~4.6 GB | `Q2_K` | ⭐ | Extreme compression, significant quality loss. | +| `qwen3-8b-claude-agentic-fable5-Q3_K_L.gguf` | 4.13 GB | ~5.6 GB | `Q3_K_L` | ⭐⭐⭐ | Slightly better than Q3_K_M, still a compromise. | +| `qwen3-8b-claude-agentic-fable5-Q3_K_M.gguf` | 3.84 GB | ~5.3 GB | `Q3_K_M` | ⭐⭐⭐ | Very small file. Quality drop noticeable. | +| `qwen3-8b-claude-agentic-fable5-Q3_K_S.gguf` | 3.51 GB | ~5.0 GB | `Q3_K_S` | ⭐⭐ | Very high compression, high quality loss. | +| `qwen3-8b-claude-agentic-fable5-Q4_K_M.gguf` | 4.68 GB | ~6.2 GB | `Q4_K_M` ✅ **Recommended** | ⭐⭐⭐⭐ | Best balance of size and quality. Recommended for most users. | +| `qwen3-8b-claude-agentic-fable5-Q4_K_S.gguf` | 4.47 GB | ~6.0 GB | `Q4_K_S` | ⭐⭐⭐½ | Good speed/size balance, slight quality loss. | +| `qwen3-8b-claude-agentic-fable5-Q5_K_M.gguf` | 5.45 GB | ~6.9 GB | `Q5_K_M` | ⭐⭐⭐⭐½ | Better quality than Q4, slightly larger. Great if you have the RAM. | +| `qwen3-8b-claude-agentic-fable5-Q5_K_S.gguf` | 5.33 GB | ~6.8 GB | `Q5_K_S` | ⭐⭐⭐⭐ | Large but accurate. | +| `qwen3-8b-claude-agentic-fable5-Q6_K.gguf` | 6.26 GB | ~7.8 GB | `Q6_K` | ⭐⭐⭐⭐⭐ | Near-perfect quality, very large. | +| `qwen3-8b-claude-agentic-fable5-Q8_0.gguf` | 8.11 GB | ~9.6 GB | `Q8_0` | ⭐⭐⭐⭐⭐ | Closest to original quality. Use when RAM is not a concern. | + + +### 💡 Which file should I download? + +- **Most users:** `qwen3-8b-claude-agentic-fable5-Q4_K_M.gguf` — best balance of size and quality +- **High RAM (32GB+):** `qwen3-8b-claude-agentic-fable5-Q8_0.gguf` — near-original quality +- **Low RAM (8GB):** `qwen3-8b-claude-agentic-fable5-Q3_K_M.gguf` — fits in 8GB with room to spare + +--- + +## ⚡ Speed Benchmarks + +*Run `python benchmark.py --model qwen3-8b-claude-agentic-fable5` to generate speed results.* + + +--- + +## 🧠 Quality Benchmarks + +*Run `kaggle_bench.ipynb` on Kaggle to benchmark this model.* + + +--- + +## 🚀 How to Use + +### Ollama +```bash +ollama run dhruvallabs/qwen3-8b-claude-agentic-fable5 +``` + +### LM Studio / Jan / Open WebUI +Search for `DhruvalLabs/qwen3-8b-claude-agentic-fable5` in the model browser. + +### llama.cpp CLI +```bash +# Download the binary from https://github.com/ggerganov/llama.cpp/releases +./llama-cli \ + -m qwen3-8b-claude-agentic-fable5-Q4_K_M.gguf \ + -p "You are a helpful assistant." \ + --conversation \ + -n 512 +``` + +### Python — llama-cpp-python +```python +from llama_cpp import Llama + +llm = Llama( + model_path="./qwen3-8b-claude-agentic-fable5-Q4_K_M.gguf", + n_gpu_layers=-1, # -1 = offload everything to GPU + n_ctx=4096, +) + +response = llm.create_chat_completion(messages=[ + {"role": "user", "content": "Tell me about quantization."} +]) +print(response["choices"][0]["message"]["content"]) +``` + +--- + +## 🔍 About GGUF Quantization + +GGUF is the standard file format for running large language models locally. +Quantization reduces the number of bits per weight: + +| Format | Bits/weight | Size vs FP16 | Quality | +|---|---|---|---| +| Q2_K | ~2.6 | 16% | ⭐ | +| Q3_K_M | ~3.3 | 21% | ⭐⭐⭐ | +| Q4_K_M | ~4.5 | 28% | ⭐⭐⭐⭐ ← sweet spot | +| Q5_K_M | ~5.6 | 35% | ⭐⭐⭐⭐½ | +| Q8_0 | ~8.5 | 53% | ⭐⭐⭐⭐⭐ | + +--- + +## 💬 Community & Feedback + +Found an issue? Have a question? Open a **Discussion** in the Community tab above. + +If these quantizations were useful, please consider: +- ⭐ Starring [quant-kit](https://github.com/DhruvalPtl/quant-kit) on GitHub +- 👍 Liking this model on HuggingFace +- 💬 Leaving feedback in the Community tab \ No newline at end of file diff --git a/qwen3-8b-claude-agentic-fable5-Q2_K.gguf b/qwen3-8b-claude-agentic-fable5-Q2_K.gguf new file mode 100644 index 0000000..cef1bec --- /dev/null +++ b/qwen3-8b-claude-agentic-fable5-Q2_K.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:66962c6852515ea5a52640acf03df8816b8c4e0f777a3b67a8497556ed48addc +size 3281733088 diff --git a/qwen3-8b-claude-agentic-fable5-Q3_K_L.gguf b/qwen3-8b-claude-agentic-fable5-Q3_K_L.gguf new file mode 100644 index 0000000..25a3481 --- /dev/null +++ b/qwen3-8b-claude-agentic-fable5-Q3_K_L.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e0cfe538420ff900aa00097ac944f17c4277c79355199e126b384fdc6e967018 +size 4431394272 diff --git a/qwen3-8b-claude-agentic-fable5-Q3_K_M.gguf b/qwen3-8b-claude-agentic-fable5-Q3_K_M.gguf new file mode 100644 index 0000000..0dc135c --- /dev/null +++ b/qwen3-8b-claude-agentic-fable5-Q3_K_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:14c0757d49cf7104ce6648c5562502cad7c9fc10390934515b5aa2a22e2ccf3d +size 4124161504 diff --git a/qwen3-8b-claude-agentic-fable5-Q3_K_S.gguf b/qwen3-8b-claude-agentic-fable5-Q3_K_S.gguf new file mode 100644 index 0000000..b2bc125 --- /dev/null +++ b/qwen3-8b-claude-agentic-fable5-Q3_K_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:d19a09b759b1c90ff1d50f4b107cd4900232f57b9bec748c6810263b10c7001c +size 3769611744 diff --git a/qwen3-8b-claude-agentic-fable5-Q4_K_M.gguf b/qwen3-8b-claude-agentic-fable5-Q4_K_M.gguf new file mode 100644 index 0000000..5afa1b9 --- /dev/null +++ b/qwen3-8b-claude-agentic-fable5-Q4_K_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c08e00c50e731c53aa2b74528675ab06a5b873ebdb36ef04dcfbe26326814340 +size 5027784160 diff --git a/qwen3-8b-claude-agentic-fable5-Q4_K_S.gguf b/qwen3-8b-claude-agentic-fable5-Q4_K_S.gguf new file mode 100644 index 0000000..0150312 --- /dev/null +++ b/qwen3-8b-claude-agentic-fable5-Q4_K_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:33eb94bb8fca60a0684ce4047ad89eef95a769c970e933399ee5287863d94df6 +size 4802012640 diff --git a/qwen3-8b-claude-agentic-fable5-Q5_K_M.gguf b/qwen3-8b-claude-agentic-fable5-Q5_K_M.gguf new file mode 100644 index 0000000..11e5f6f --- /dev/null +++ b/qwen3-8b-claude-agentic-fable5-Q5_K_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:75b25d7273f645d57f5b8141cc3c767e4a742aa90ce52adcb84e4d0030ddf9e0 +size 5851112928 diff --git a/qwen3-8b-claude-agentic-fable5-Q5_K_S.gguf b/qwen3-8b-claude-agentic-fable5-Q5_K_S.gguf new file mode 100644 index 0000000..e58635b --- /dev/null +++ b/qwen3-8b-claude-agentic-fable5-Q5_K_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:399e08917f9ae57da12659d962a164bc74eb54158fd928a58d02fb874e292e13 +size 5720761824 diff --git a/qwen3-8b-claude-agentic-fable5-Q6_K.gguf b/qwen3-8b-claude-agentic-fable5-Q6_K.gguf new file mode 100644 index 0000000..c10bc7d --- /dev/null +++ b/qwen3-8b-claude-agentic-fable5-Q6_K.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:1f3e7e7ec69c1c2ac90dfda29801ebdecf14d09ec219c598ae518cd6cd4990ea +size 6725899744 diff --git a/qwen3-8b-claude-agentic-fable5-Q8_0.gguf b/qwen3-8b-claude-agentic-fable5-Q8_0.gguf new file mode 100644 index 0000000..e003fb3 --- /dev/null +++ b/qwen3-8b-claude-agentic-fable5-Q8_0.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:aa13c8cb80e2de28a4617216f40eabb6d17148fe53d2c04b2991457d776557e2 +size 8709518816