Granite-4.1-8B-SFT-Claude-Opus-Reasoning-Unsloth-GGUF/README.md

---
license: apache-2.0
language:
  - en
library_name: gguf
pipeline_tag: text-generation
tags:
  - gguf
  - llama.cpp
  - quantized
  - granite
  - reasoning
  - unsloth
  - text-generation
base_model: ermiaazarkhalili/Granite-4.1-8B-SFT-Claude-Opus-Reasoning-Unsloth
datasets:
  - ermiaazarkhalili/claude-reasoning-distillation
---

# Granite-4.1-8B-SFT-Claude-Opus-Reasoning-Unsloth-GGUF

GGUF quantizations of [Granite-4.1-8B-SFT-Claude-Opus-Reasoning-Unsloth](https://huggingface.co/ermiaazarkhalili/Granite-4.1-8B-SFT-Claude-Opus-Reasoning-Unsloth) for **CPU and edge inference** with [llama.cpp](https://github.com/ggerganov/llama.cpp), [Ollama](https://ollama.com), LM Studio, and other GGUF runtimes.

This model is a fine-tune of [Granite 4.1-8B](https://huggingface.co/ibm-granite/granite-4.1-8b) for **reasoning distillation (chain-of-thought)**, trained with [Unsloth](https://github.com/unslothai/unsloth) on [claude-reasoning-distillation](https://huggingface.co/datasets/ermiaazarkhalili/claude-reasoning-distillation).

- **Full-precision model:** [Granite-4.1-8B-SFT-Claude-Opus-Reasoning-Unsloth](https://huggingface.co/ermiaazarkhalili/Granite-4.1-8B-SFT-Claude-Opus-Reasoning-Unsloth)
- **Base model:** [Granite 4.1-8B](https://huggingface.co/ibm-granite/granite-4.1-8b)
- **Parameters:** 8B

## Available Quantizations

| Quant | Size | Recommended Use | File |
|-------|------|-----------------|------|
| `Q2_K` | 3.41 GB | Smallest, lowest quality — quick tests / very constrained devices | [`granite-4.1-8b-sft-claude-opus-reasoning-unsloth.q2_k.gguf`](https://huggingface.co/ermiaazarkhalili/Granite-4.1-8B-SFT-Claude-Opus-Reasoning-Unsloth-GGUF/blob/main/granite-4.1-8b-sft-claude-opus-reasoning-unsloth.q2_k.gguf) |
| `Q3_K_M` | 4.35 GB | Small, acceptable quality | [`granite-4.1-8b-sft-claude-opus-reasoning-unsloth.q3_k_m.gguf`](https://huggingface.co/ermiaazarkhalili/Granite-4.1-8B-SFT-Claude-Opus-Reasoning-Unsloth-GGUF/blob/main/granite-4.1-8b-sft-claude-opus-reasoning-unsloth.q3_k_m.gguf) |
| `Q4_K_M` | 5.35 GB | Recommended — best size/quality balance | [`granite-4.1-8b-sft-claude-opus-reasoning-unsloth.q4_k_m.gguf`](https://huggingface.co/ermiaazarkhalili/Granite-4.1-8B-SFT-Claude-Opus-Reasoning-Unsloth-GGUF/blob/main/granite-4.1-8b-sft-claude-opus-reasoning-unsloth.q4_k_m.gguf) |
| `Q5_K_M` | 6.25 GB | High quality, larger | [`granite-4.1-8b-sft-claude-opus-reasoning-unsloth.q5_k_m.gguf`](https://huggingface.co/ermiaazarkhalili/Granite-4.1-8B-SFT-Claude-Opus-Reasoning-Unsloth-GGUF/blob/main/granite-4.1-8b-sft-claude-opus-reasoning-unsloth.q5_k_m.gguf) |
| `Q6_K` | 7.22 GB | Very high quality, near-fp16 | [`granite-4.1-8b-sft-claude-opus-reasoning-unsloth.q6_k.gguf`](https://huggingface.co/ermiaazarkhalili/Granite-4.1-8B-SFT-Claude-Opus-Reasoning-Unsloth-GGUF/blob/main/granite-4.1-8b-sft-claude-opus-reasoning-unsloth.q6_k.gguf) |
| `Q8_0` | 9.35 GB | Near-lossless, largest | [`granite-4.1-8b-sft-claude-opus-reasoning-unsloth.q8_0.gguf`](https://huggingface.co/ermiaazarkhalili/Granite-4.1-8B-SFT-Claude-Opus-Reasoning-Unsloth-GGUF/blob/main/granite-4.1-8b-sft-claude-opus-reasoning-unsloth.q8_0.gguf) |

`Q4_K_M` is the recommended default for most users.

## Usage

### Download a single quant

```bash
pip install -U "huggingface_hub[cli]"
hf download ermiaazarkhalili/Granite-4.1-8B-SFT-Claude-Opus-Reasoning-Unsloth-GGUF \
  --include "*q4_k_m*.gguf" --local-dir ./Granite-4.1-8B-SFT-Claude-Opus-Reasoning-Unsloth-GGUF
```

### llama.cpp

```bash
# build: https://github.com/ggerganov/llama.cpp
./llama-cli -m ./Granite-4.1-8B-SFT-Claude-Opus-Reasoning-Unsloth-GGUF/granite-4.1-8b-sft-claude-opus-reasoning-unsloth.q2_k.gguf \
  -p "Solve step by step: What is the sum of the first 10 prime numbers?" -n 512
```

### Ollama

```bash
ollama run hf.co/ermiaazarkhalili/Granite-4.1-8B-SFT-Claude-Opus-Reasoning-Unsloth-GGUF:Q4_K_M "Solve step by step: What is the sum of the first 10 prime numbers?"
```

## Training Outcome

| Metric | Value |
|--------|-------|
| SLURM Job ID | `38330897` |
| Runtime | 32m 49s |
| Final Training Loss | 0.7895 |
| Peak VRAM | 9.26 GB |
| GPU | H100 80GB HBM3 (MIG 3g.40gb) |

## License

APACHE-2.0 — see the [base model](https://huggingface.co/ibm-granite/granite-4.1-8b) for full terms.

## Acknowledgments

- [Unsloth](https://github.com/unslothai/unsloth) for 2x faster fine-tuning
- [llama.cpp](https://github.com/ggerganov/llama.cpp) for GGUF quantization
- Base model developers (ibm-granite)
- [Compute Canada / DRAC](https://alliancecan.ca/) for HPC resources