SmolLM3-3B-Instruct-GGUF/README.md

---
license: apache-2.0
license_link: https://www.apache.org/licenses/LICENSE-2.0
base_model: HuggingFaceTB/SmolLM3-3B
library_name: gguf
pipeline_tag: text-generation
language:
  - en
  - fr
  - es
  - de
  - it
  - pt
tags:
  - gguf
  - llama.cpp
  - quantized
  - on-device
  - mobile
  - apple-silicon
  - haplo
  - smollm3
inference: false
---

# SmolLM3-3B — GGUF (iPhone-optimized)

GGUF quantizations of [`HuggingFaceTB/SmolLM3-3B`](https://huggingface.co/HuggingFaceTB/SmolLM3-3B), built and optimized for on-device inference on iPhone, iPad, and Apple Silicon Macs via [llama.cpp](https://github.com/ggerganov/llama.cpp) or apps that wrap it (e.g. [Haplo](https://haploapp.com)).

> Built and quantized by **jc-builds** for the [Haplo](https://haploapp.com) ecosystem. Original weights © Hugging Face, redistributed under Apache 2.0 per the upstream license.

## TL;DR

A 3B-parameter decoder-only transformer with **hybrid reasoning** (toggle "thinking mode" via `/think` or `/no_think` system prompts), **128k context** (with YaRN), and 6 native languages. SmolLM3 is the rare model where **everything is open** — weights, training data mixture, and training configs. At the 3B scale it outperforms Llama-3.2-3B and Qwen2.5-3B across most benchmarks and stays competitive with many 4B-class models.

## Available quantizations

| File | Size | Bits/weight | Recommended use |
|------|------|-------------|-----------------|
| `SmolLM3-3B-Q4_K_M.gguf` | 1.8 GB | 4.8 | **Default — best size/quality tradeoff for phone & laptop** |
| `SmolLM3-3B-Q5_K_M.gguf` | 2.1 GB | 5.7 | Slightly better quality, ~17% bigger; good for iPad / Mac |
| `SmolLM3-3B-Q8_0.gguf` | 3.0 GB | 8.5 | Near-FP16 quality; only worth it on Apple Silicon Mac |

**Pick `Q4_K_M`** unless you have a reason not to — it's the sweet spot for on-device on Apple Silicon. Q5_K_M is ~5-10% smarter on hard reasoning prompts but ~20% bigger; Q8_0 is essentially indistinguishable from FP16 but 2× the size of Q4_K_M.

## Performance on Apple Silicon

Approximate decode throughput at single-batch greedy decode, 2048-token context. Measured with `llama-cli`.

| Device | RAM | Q4_K_M tok/s | Notes |
|--------|-----|--------------|-------|
| iPhone 15 Pro | 8 GB | ~22 tok/s | Smooth chat experience |
| iPhone 14 Pro | 6 GB | ~18 tok/s | Comfortable |
| iPad Pro M2 | 8 GB | ~45 tok/s | Snappy |
| MacBook Pro M3 | 16 GB | ~80 tok/s | Effectively instant |

> Reference numbers — your throughput will vary with prompt length, KV cache, and what else is running. Q5_K_M and Q8_0 are roughly 15% / 40% slower than Q4_K_M respectively.

## How to use

### 1. Haplo (iPhone / iPad / Mac)

The model appears automatically in Haplo's model browser on Kuzco-1.1.0+ builds. The download URL for Q4_K_M is:

```
https://huggingface.co/jc-builds/SmolLM3-3B-Instruct-GGUF/resolve/main/SmolLM3-3B-Q4_K_M.gguf
```

### 2. llama.cpp (CLI)

```bash
huggingface-cli download jc-builds/SmolLM3-3B-Instruct-GGUF SmolLM3-3B-Q4_K_M.gguf --local-dir .

./llama-cli \
  -m SmolLM3-3B-Q4_K_M.gguf \
  -p "Explain gravity in two sentences." \
  -n 256 \
  --temp 0.6 \
  --top-p 0.95
```

### 3. Ollama

```bash
cat <<'EOF' > Modelfile
FROM ./SmolLM3-3B-Q4_K_M.gguf
PARAMETER temperature 0.6
PARAMETER top_p 0.95
EOF
ollama create smollm3 -f Modelfile
ollama run smollm3
```

## Reasoning modes (think / no_think)

SmolLM3 ships with hybrid reasoning. You toggle it via system prompt:

| System prompt | Behavior |
|---|---|
| `/think` (default) | Emits a `<think>…</think>` reasoning block, then the answer. Better on math / code / multi-step problems. |
| `/no_think` | Skips the reasoning block. Use for fast chat / simple Q&A. |

Example:
```
<|im_start|>system
/no_think<|im_end|>
<|im_start|>user
Capital of Australia?<|im_end|>
<|im_start|>assistant
```

## Sampling defaults

The upstream team recommends `temperature=0.6` and `top_p=0.95`. The GGUF metadata stores these as the defaults — most clients (llama.cpp, Haplo, Ollama) will use them automatically.

## Chat template

The HuggingFaceTB chat template is preserved in the GGUF metadata (so llama.cpp's `--chat-template` flag is *not* required). It uses ChatML-style turns:

```
<|im_start|>system
{system}<|im_end|>
<|im_start|>user
{user}<|im_end|>
<|im_start|>assistant
{assistant}<|im_end|>
```

## Quantization recipe

Built with [llama.cpp](https://github.com/ggerganov/llama.cpp) at commit `e43431b` (May 7, 2026).

1. Downloaded `HuggingFaceTB/SmolLM3-3B` safetensors checkpoint via `huggingface-cli`.
2. Converted to GGUF FP16 via `convert_hf_to_gguf.py --outtype f16`.
3. Quantized to each target type via `llama-quantize`:
   ```bash
   llama-quantize SmolLM3-3B-F16.gguf SmolLM3-3B-Q4_K_M.gguf Q4_K_M
   llama-quantize SmolLM3-3B-F16.gguf SmolLM3-3B-Q5_K_M.gguf Q5_K_M
   llama-quantize SmolLM3-3B-F16.gguf SmolLM3-3B-Q8_0.gguf   Q8_0
   ```

No imatrix calibration was used — the weights come from the upstream FP16 directly.

## Original model card

See the upstream model card for full architecture, training, and benchmark details: [HuggingFaceTB/SmolLM3-3B](https://huggingface.co/HuggingFaceTB/SmolLM3-3B).

## License

Apache 2.0, inherited from the original model. Commercial use, modification, and redistribution are permitted. See [LICENSE](https://www.apache.org/licenses/LICENSE-2.0) for the full terms.

> SmolLM3 by Hugging Face. Licensed under Apache 2.0.

## Acknowledgements

- The [Hugging Face SmolLM team](https://huggingface.co/HuggingFaceTB) for the original weights and an unusually generous open-everything release (training data, recipe, configs).
- The [llama.cpp](https://github.com/ggerganov/llama.cpp) team for the GGUF format and quantization tooling.
- The [Haplo](https://haploapp.com) ecosystem this drop is built for.