From 32d8ae16b1cc4f8cfb454e3fcb8c39cf8167b233 Mon Sep 17 00:00:00 2001 From: ModelHub XC Date: Sun, 14 Jun 2026 15:14:18 +0800 Subject: [PATCH] =?UTF-8?q?=E5=88=9D=E5=A7=8B=E5=8C=96=E9=A1=B9=E7=9B=AE?= =?UTF-8?q?=EF=BC=8C=E7=94=B1ModelHub=20XC=E7=A4=BE=E5=8C=BA=E6=8F=90?= =?UTF-8?q?=E4=BE=9B=E6=A8=A1=E5=9E=8B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Model: jc-builds/SmolLM3-3B-Instruct-GGUF Source: Original Platform --- .gitattributes | 38 ++++++++++ README.md | 158 +++++++++++++++++++++++++++++++++++++++++ SmolLM3-3B-Q4_K_M.gguf | 3 + SmolLM3-3B-Q5_K_M.gguf | 3 + SmolLM3-3B-Q8_0.gguf | 3 + 5 files changed, 205 insertions(+) create mode 100644 .gitattributes create mode 100644 README.md create mode 100644 SmolLM3-3B-Q4_K_M.gguf create mode 100644 SmolLM3-3B-Q5_K_M.gguf create mode 100644 SmolLM3-3B-Q8_0.gguf diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..e05d606 --- /dev/null +++ b/.gitattributes @@ -0,0 +1,38 @@ +*.7z filter=lfs diff=lfs merge=lfs -text +*.arrow filter=lfs diff=lfs merge=lfs -text +*.bin filter=lfs diff=lfs merge=lfs -text +*.bz2 filter=lfs diff=lfs merge=lfs -text +*.ckpt filter=lfs diff=lfs merge=lfs -text +*.ftz filter=lfs diff=lfs merge=lfs -text +*.gz filter=lfs diff=lfs merge=lfs -text +*.h5 filter=lfs diff=lfs merge=lfs -text +*.joblib filter=lfs diff=lfs merge=lfs -text +*.lfs.* filter=lfs diff=lfs merge=lfs -text +*.mlmodel filter=lfs diff=lfs merge=lfs -text +*.model filter=lfs diff=lfs merge=lfs -text +*.msgpack filter=lfs diff=lfs merge=lfs -text +*.npy filter=lfs diff=lfs merge=lfs -text +*.npz filter=lfs diff=lfs merge=lfs -text +*.onnx filter=lfs diff=lfs merge=lfs -text +*.ot filter=lfs diff=lfs merge=lfs -text +*.parquet filter=lfs diff=lfs merge=lfs -text +*.pb filter=lfs diff=lfs merge=lfs -text +*.pickle filter=lfs diff=lfs merge=lfs -text +*.pkl filter=lfs diff=lfs merge=lfs -text +*.pt filter=lfs diff=lfs merge=lfs -text +*.pth filter=lfs diff=lfs merge=lfs -text +*.rar filter=lfs diff=lfs merge=lfs -text +*.safetensors filter=lfs diff=lfs merge=lfs -text +saved_model/**/* filter=lfs diff=lfs merge=lfs -text +*.tar.* filter=lfs diff=lfs merge=lfs -text +*.tar filter=lfs diff=lfs merge=lfs -text +*.tflite filter=lfs diff=lfs merge=lfs -text +*.tgz filter=lfs diff=lfs merge=lfs -text +*.wasm filter=lfs diff=lfs merge=lfs -text +*.xz filter=lfs diff=lfs merge=lfs -text +*.zip filter=lfs diff=lfs merge=lfs -text +*.zst filter=lfs diff=lfs merge=lfs -text +*tfevents* filter=lfs diff=lfs merge=lfs -text +SmolLM3-3B-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text +SmolLM3-3B-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text +SmolLM3-3B-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text diff --git a/README.md b/README.md new file mode 100644 index 0000000..be9dfd4 --- /dev/null +++ b/README.md @@ -0,0 +1,158 @@ +--- +license: apache-2.0 +license_link: https://www.apache.org/licenses/LICENSE-2.0 +base_model: HuggingFaceTB/SmolLM3-3B +library_name: gguf +pipeline_tag: text-generation +language: + - en + - fr + - es + - de + - it + - pt +tags: + - gguf + - llama.cpp + - quantized + - on-device + - mobile + - apple-silicon + - haplo + - smollm3 +inference: false +--- + +# SmolLM3-3B — GGUF (iPhone-optimized) + +GGUF quantizations of [`HuggingFaceTB/SmolLM3-3B`](https://huggingface.co/HuggingFaceTB/SmolLM3-3B), built and optimized for on-device inference on iPhone, iPad, and Apple Silicon Macs via [llama.cpp](https://github.com/ggerganov/llama.cpp) or apps that wrap it (e.g. [Haplo](https://haploapp.com)). + +> Built and quantized by **jc-builds** for the [Haplo](https://haploapp.com) ecosystem. Original weights © Hugging Face, redistributed under Apache 2.0 per the upstream license. + +## TL;DR + +A 3B-parameter decoder-only transformer with **hybrid reasoning** (toggle "thinking mode" via `/think` or `/no_think` system prompts), **128k context** (with YaRN), and 6 native languages. SmolLM3 is the rare model where **everything is open** — weights, training data mixture, and training configs. At the 3B scale it outperforms Llama-3.2-3B and Qwen2.5-3B across most benchmarks and stays competitive with many 4B-class models. + +## Available quantizations + +| File | Size | Bits/weight | Recommended use | +|------|------|-------------|-----------------| +| `SmolLM3-3B-Q4_K_M.gguf` | 1.8 GB | 4.8 | **Default — best size/quality tradeoff for phone & laptop** | +| `SmolLM3-3B-Q5_K_M.gguf` | 2.1 GB | 5.7 | Slightly better quality, ~17% bigger; good for iPad / Mac | +| `SmolLM3-3B-Q8_0.gguf` | 3.0 GB | 8.5 | Near-FP16 quality; only worth it on Apple Silicon Mac | + +**Pick `Q4_K_M`** unless you have a reason not to — it's the sweet spot for on-device on Apple Silicon. Q5_K_M is ~5-10% smarter on hard reasoning prompts but ~20% bigger; Q8_0 is essentially indistinguishable from FP16 but 2× the size of Q4_K_M. + +## Performance on Apple Silicon + +Approximate decode throughput at single-batch greedy decode, 2048-token context. Measured with `llama-cli`. + +| Device | RAM | Q4_K_M tok/s | Notes | +|--------|-----|--------------|-------| +| iPhone 15 Pro | 8 GB | ~22 tok/s | Smooth chat experience | +| iPhone 14 Pro | 6 GB | ~18 tok/s | Comfortable | +| iPad Pro M2 | 8 GB | ~45 tok/s | Snappy | +| MacBook Pro M3 | 16 GB | ~80 tok/s | Effectively instant | + +> Reference numbers — your throughput will vary with prompt length, KV cache, and what else is running. Q5_K_M and Q8_0 are roughly 15% / 40% slower than Q4_K_M respectively. + +## How to use + +### 1. Haplo (iPhone / iPad / Mac) + +The model appears automatically in Haplo's model browser on Kuzco-1.1.0+ builds. The download URL for Q4_K_M is: + +``` +https://huggingface.co/jc-builds/SmolLM3-3B-Instruct-GGUF/resolve/main/SmolLM3-3B-Q4_K_M.gguf +``` + +### 2. llama.cpp (CLI) + +```bash +huggingface-cli download jc-builds/SmolLM3-3B-Instruct-GGUF SmolLM3-3B-Q4_K_M.gguf --local-dir . + +./llama-cli \ + -m SmolLM3-3B-Q4_K_M.gguf \ + -p "Explain gravity in two sentences." \ + -n 256 \ + --temp 0.6 \ + --top-p 0.95 +``` + +### 3. Ollama + +```bash +cat <<'EOF' > Modelfile +FROM ./SmolLM3-3B-Q4_K_M.gguf +PARAMETER temperature 0.6 +PARAMETER top_p 0.95 +EOF +ollama create smollm3 -f Modelfile +ollama run smollm3 +``` + +## Reasoning modes (think / no_think) + +SmolLM3 ships with hybrid reasoning. You toggle it via system prompt: + +| System prompt | Behavior | +|---|---| +| `/think` (default) | Emits a `…` reasoning block, then the answer. Better on math / code / multi-step problems. | +| `/no_think` | Skips the reasoning block. Use for fast chat / simple Q&A. | + +Example: +``` +<|im_start|>system +/no_think<|im_end|> +<|im_start|>user +Capital of Australia?<|im_end|> +<|im_start|>assistant +``` + +## Sampling defaults + +The upstream team recommends `temperature=0.6` and `top_p=0.95`. The GGUF metadata stores these as the defaults — most clients (llama.cpp, Haplo, Ollama) will use them automatically. + +## Chat template + +The HuggingFaceTB chat template is preserved in the GGUF metadata (so llama.cpp's `--chat-template` flag is *not* required). It uses ChatML-style turns: + +``` +<|im_start|>system +{system}<|im_end|> +<|im_start|>user +{user}<|im_end|> +<|im_start|>assistant +{assistant}<|im_end|> +``` + +## Quantization recipe + +Built with [llama.cpp](https://github.com/ggerganov/llama.cpp) at commit `e43431b` (May 7, 2026). + +1. Downloaded `HuggingFaceTB/SmolLM3-3B` safetensors checkpoint via `huggingface-cli`. +2. Converted to GGUF FP16 via `convert_hf_to_gguf.py --outtype f16`. +3. Quantized to each target type via `llama-quantize`: + ```bash + llama-quantize SmolLM3-3B-F16.gguf SmolLM3-3B-Q4_K_M.gguf Q4_K_M + llama-quantize SmolLM3-3B-F16.gguf SmolLM3-3B-Q5_K_M.gguf Q5_K_M + llama-quantize SmolLM3-3B-F16.gguf SmolLM3-3B-Q8_0.gguf Q8_0 + ``` + +No imatrix calibration was used — the weights come from the upstream FP16 directly. + +## Original model card + +See the upstream model card for full architecture, training, and benchmark details: [HuggingFaceTB/SmolLM3-3B](https://huggingface.co/HuggingFaceTB/SmolLM3-3B). + +## License + +Apache 2.0, inherited from the original model. Commercial use, modification, and redistribution are permitted. See [LICENSE](https://www.apache.org/licenses/LICENSE-2.0) for the full terms. + +> SmolLM3 by Hugging Face. Licensed under Apache 2.0. + +## Acknowledgements + +- The [Hugging Face SmolLM team](https://huggingface.co/HuggingFaceTB) for the original weights and an unusually generous open-everything release (training data, recipe, configs). +- The [llama.cpp](https://github.com/ggerganov/llama.cpp) team for the GGUF format and quantization tooling. +- The [Haplo](https://haploapp.com) ecosystem this drop is built for. diff --git a/SmolLM3-3B-Q4_K_M.gguf b/SmolLM3-3B-Q4_K_M.gguf new file mode 100644 index 0000000..482e353 --- /dev/null +++ b/SmolLM3-3B-Q4_K_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:233063e030ac4159d5bca4e1a8b8f587bd625efd2a46dbc72eceb0628acdbe3f +size 1915305888 diff --git a/SmolLM3-3B-Q5_K_M.gguf b/SmolLM3-3B-Q5_K_M.gguf new file mode 100644 index 0000000..45716dd --- /dev/null +++ b/SmolLM3-3B-Q5_K_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:4a6e39157ee99fec3048717a3d19ebf2dceac7eabab33702528aa9f9e8c90f07 +size 2213756832 diff --git a/SmolLM3-3B-Q8_0.gguf b/SmolLM3-3B-Q8_0.gguf new file mode 100644 index 0000000..db3e67f --- /dev/null +++ b/SmolLM3-3B-Q8_0.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:4e6378d678a215e410b7233fe818d72585710aec78d6b7808be3f46eb96887eb +size 3275575200