a0b0c6e4c9
Model: ermiaazarkhalili/Qwen3-8B-SFT-Claude-Opus-Reasoning-Unsloth-GGUF Source: Original Platform
87 lines
4.3 KiB
Markdown
87 lines
4.3 KiB
Markdown
---
|
|
license: apache-2.0
|
|
language:
|
|
- en
|
|
library_name: gguf
|
|
pipeline_tag: text-generation
|
|
tags:
|
|
- gguf
|
|
- llama.cpp
|
|
- quantized
|
|
- qwen3
|
|
- reasoning
|
|
- unsloth
|
|
- text-generation
|
|
base_model: ermiaazarkhalili/Qwen3-8B-SFT-Claude-Opus-Reasoning-Unsloth
|
|
datasets:
|
|
- ermiaazarkhalili/claude-reasoning-distillation
|
|
---
|
|
|
|
# Qwen3-8B-SFT-Claude-Opus-Reasoning-Unsloth-GGUF
|
|
|
|
GGUF quantizations of [Qwen3-8B-SFT-Claude-Opus-Reasoning-Unsloth](https://huggingface.co/ermiaazarkhalili/Qwen3-8B-SFT-Claude-Opus-Reasoning-Unsloth) for **CPU and edge inference** with [llama.cpp](https://github.com/ggerganov/llama.cpp), [Ollama](https://ollama.com), LM Studio, and other GGUF runtimes.
|
|
|
|
This model is a fine-tune of [Qwen3-8B (Unsloth 4-bit)](https://huggingface.co/unsloth/qwen3-8b-unsloth-bnb-4bit) for **reasoning distillation (chain-of-thought)**, trained with [Unsloth](https://github.com/unslothai/unsloth) on [claude-reasoning-distillation](https://huggingface.co/datasets/ermiaazarkhalili/claude-reasoning-distillation).
|
|
|
|
- **Full-precision model:** [Qwen3-8B-SFT-Claude-Opus-Reasoning-Unsloth](https://huggingface.co/ermiaazarkhalili/Qwen3-8B-SFT-Claude-Opus-Reasoning-Unsloth)
|
|
- **Base model:** [Qwen3-8B (Unsloth 4-bit)](https://huggingface.co/unsloth/qwen3-8b-unsloth-bnb-4bit)
|
|
- **Parameters:** 8B
|
|
|
|
## Available Quantizations
|
|
|
|
| Quant | Size | Recommended Use | File |
|
|
|-------|------|-----------------|------|
|
|
| `Q2_K` | 3.28 GB | Smallest, lowest quality — quick tests / very constrained devices | [`qwen3-8b-sft-claude-opus-reasoning-unsloth.q2_k.gguf`](https://huggingface.co/ermiaazarkhalili/Qwen3-8B-SFT-Claude-Opus-Reasoning-Unsloth-GGUF/blob/main/qwen3-8b-sft-claude-opus-reasoning-unsloth.q2_k.gguf) |
|
|
| `Q3_K_M` | 4.12 GB | Small, acceptable quality | [`qwen3-8b-sft-claude-opus-reasoning-unsloth.q3_k_m.gguf`](https://huggingface.co/ermiaazarkhalili/Qwen3-8B-SFT-Claude-Opus-Reasoning-Unsloth-GGUF/blob/main/qwen3-8b-sft-claude-opus-reasoning-unsloth.q3_k_m.gguf) |
|
|
| `Q4_K_M` | 5.03 GB | Recommended — best size/quality balance | [`qwen3-8b-sft-claude-opus-reasoning-unsloth.q4_k_m.gguf`](https://huggingface.co/ermiaazarkhalili/Qwen3-8B-SFT-Claude-Opus-Reasoning-Unsloth-GGUF/blob/main/qwen3-8b-sft-claude-opus-reasoning-unsloth.q4_k_m.gguf) |
|
|
| `Q5_K_M` | 5.85 GB | High quality, larger | [`qwen3-8b-sft-claude-opus-reasoning-unsloth.q5_k_m.gguf`](https://huggingface.co/ermiaazarkhalili/Qwen3-8B-SFT-Claude-Opus-Reasoning-Unsloth-GGUF/blob/main/qwen3-8b-sft-claude-opus-reasoning-unsloth.q5_k_m.gguf) |
|
|
| `Q6_K` | 6.73 GB | Very high quality, near-fp16 | [`qwen3-8b-sft-claude-opus-reasoning-unsloth.q6_k.gguf`](https://huggingface.co/ermiaazarkhalili/Qwen3-8B-SFT-Claude-Opus-Reasoning-Unsloth-GGUF/blob/main/qwen3-8b-sft-claude-opus-reasoning-unsloth.q6_k.gguf) |
|
|
| `Q8_0` | 8.71 GB | Near-lossless, largest | [`qwen3-8b-sft-claude-opus-reasoning-unsloth.q8_0.gguf`](https://huggingface.co/ermiaazarkhalili/Qwen3-8B-SFT-Claude-Opus-Reasoning-Unsloth-GGUF/blob/main/qwen3-8b-sft-claude-opus-reasoning-unsloth.q8_0.gguf) |
|
|
|
|
`Q4_K_M` is the recommended default for most users.
|
|
|
|
## Usage
|
|
|
|
### Download a single quant
|
|
|
|
```bash
|
|
pip install -U "huggingface_hub[cli]"
|
|
hf download ermiaazarkhalili/Qwen3-8B-SFT-Claude-Opus-Reasoning-Unsloth-GGUF \
|
|
--include "*q4_k_m*.gguf" --local-dir ./Qwen3-8B-SFT-Claude-Opus-Reasoning-Unsloth-GGUF
|
|
```
|
|
|
|
### llama.cpp
|
|
|
|
```bash
|
|
# build: https://github.com/ggerganov/llama.cpp
|
|
./llama-cli -m ./Qwen3-8B-SFT-Claude-Opus-Reasoning-Unsloth-GGUF/qwen3-8b-sft-claude-opus-reasoning-unsloth.q2_k.gguf \
|
|
-p "Solve step by step: What is the sum of the first 10 prime numbers?" -n 512
|
|
```
|
|
|
|
### Ollama
|
|
|
|
```bash
|
|
ollama run hf.co/ermiaazarkhalili/Qwen3-8B-SFT-Claude-Opus-Reasoning-Unsloth-GGUF:Q4_K_M "Solve step by step: What is the sum of the first 10 prime numbers?"
|
|
```
|
|
|
|
## Training Outcome
|
|
|
|
| Metric | Value |
|
|
|--------|-------|
|
|
| SLURM Job ID | `36885901` |
|
|
| Runtime | 40m 30s |
|
|
| Final Training Loss | 0.8753 |
|
|
| Peak VRAM | 14.23 GB |
|
|
| GPU | H100 80GB HBM3 (MIG 3g.40gb) |
|
|
|
|
## License
|
|
|
|
APACHE-2.0 — see the [base model](https://huggingface.co/unsloth/qwen3-8b-unsloth-bnb-4bit) for full terms.
|
|
|
|
## Acknowledgments
|
|
|
|
- [Unsloth](https://github.com/unslothai/unsloth) for 2x faster fine-tuning
|
|
- [llama.cpp](https://github.com/ggerganov/llama.cpp) for GGUF quantization
|
|
- Base model developers (unsloth)
|
|
- [Compute Canada / DRAC](https://alliancecan.ca/) for HPC resources
|