Kepler-8B-Instruct-v2/README.md

---
license: apache-2.0
base_model: intellectlabs/Kepler-8B-Instruct-v2
tags:
- gguf
- llama.cpp
- quantized
- merge
- mergekit
- qwen2
- text-generation
- conversational
language:
- en
pipeline_tag: text-generation
---

<div align="center">

# 🪐 Kepler-8B-Instruct — GGUF

### Quantized GGUF builds of Kepler-8B-Instruct — ready to run anywhere llama.cpp does.

**License: Apache 2.0**  ·  **Format: GGUF**  ·  **Base: [Kepler-8B-Instruct](https://huggingface.co/intellectlabs/Kepler-8B-Instruct)**

**[Original Model](https://huggingface.co/intellectlabs/Kepler-8B-Instruct)** · **[llama.cpp](https://github.com/ggerganov/llama.cpp)** · **[Discussions](../../discussions)**

</div>

---

## ✨ What is this?

**Kepler-8B-Instruct** is an 8-billion-parameter assistant built by merging `Qwen/Qwen2.5-7B-Instruct` (85%) with `deepseek-ai/DeepSeek-R1-Distill-Qwen-7B` (15%) using [mergekit](https://github.com/arcee-ai/mergekit). The blend keeps Qwen2.5-Instruct's clean, reliable instruction-following as the foundation while folding in a touch of DeepSeek's reasoning-distilled behavior.

This repository packages that model as **GGUF** — the universal format for [llama.cpp](https://github.com/ggerganov/llama.cpp) and everything built on it. One file, no Python environment, no CUDA setup. It runs on a gaming laptop, a Raspberry Pi-class board, or a headless server with equal ease.

**Highlights:**
- 🧩 A reliable instruction-following foundation with a light touch of reasoning-model flavor
- ⚡ Seven precision levels included, from a ~3 GB ultra-light build to full-precision F16
- 🔌 Drop-in compatible with llama.cpp, LM Studio, Ollama, koboldcpp, GPT4All, Jan, and text-generation-webui
- 🗂️ Native ChatML support, with a built-in default identity — ask "who are you" and it answers as Kepler by intellectlabs, in whatever language you ask in

## 🚀 Quick Start

### llama.cpp
```bash
./llama-cli -hf intellectlabs/Kepler-8B-Instruct-GGUF:Q4_K_M -p "Explain quantum entanglement simply."
```

Or download a specific file manually:
```bash
huggingface-cli download intellectlabs/Kepler-8B-Instruct-GGUF Kepler-8B-Instruct-Q4_K_M.gguf --local-dir .
./llama-cli -m Kepler-8B-Instruct-Q4_K_M.gguf -p "Hello, who are you?" -cnv
```

### LM Studio
Search **`intellectlabs/Kepler-8B-Instruct-GGUF`** directly in the LM Studio model search bar and download your preferred quant.

### Ollama
```bash
ollama run hf.co/intellectlabs/Kepler-8B-Instruct-GGUF:Q4_K_M
```

## 📦 Available Quantizations

| File | Quant | Size (approx.) | Quality | Recommended For |
|---|---|---|---|---|
| `Kepler-8B-Instruct-Q2_K.gguf` | Q2_K | ~3.1 GB | ⭐️ | Extreme low-RAM devices only |
| `Kepler-8B-Instruct-Q3_K_M.gguf` | Q3_K_M | ~4.0 GB | ⭐️⭐️ | Low-RAM systems, testing |
| `Kepler-8B-Instruct-Q4_0.gguf` | Q4_0 | ~4.7 GB | ⭐️⭐️⭐️ | Legacy-hardware compatibility |
| `Kepler-8B-Instruct-Q4_K_M.gguf` | Q4_K_M | ~4.9 GB | ⭐️⭐️⭐️⭐️ | **Best balance — recommended default** |
| `Kepler-8B-Instruct-Q5_K_M.gguf` | Q5_K_M | ~5.7 GB | ⭐️⭐️⭐️⭐️ | Higher quality, still efficient |
| `Kepler-8B-Instruct-Q8_0.gguf` | Q8_0 | ~8.5 GB | ⭐️⭐️⭐️⭐️⭐️ | Near-lossless, if you have the RAM |
| `Kepler-8B-Instruct-F16.gguf` | F16 | ~16 GB | ⭐️⭐️⭐️⭐️⭐️ | Full precision, GPU/server use |

> 💡 **New here?** Start with **Q4_K_M** — it's the sweet spot most people use: small enough to run comfortably, strong enough to feel close to the full model.

## 🧠 Prompt Format

This model uses the **ChatML** template (inherited from its Qwen2 base):

```
<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
```

Most tools (llama.cpp `-cnv`, LM Studio, Ollama) apply this automatically via the embedded chat template — no manual formatting needed.

## 🛠️ How It Was Made

1. **Merge** — `deepseek-ai/DeepSeek-R1-Distill-Qwen-7B` linearly blended into `Qwen/Qwen2.5-7B-Instruct` (15%/85%) using mergekit, with the tokenizer taken from the base model.
2. **Convert** — merged safetensors converted to GGUF (F16) using `llama.cpp`'s `convert_hf_to_gguf.py`.
3. **Quantize** — F16 GGUF quantized into multiple formats using `llama-quantize`, covering everything from ultra-compact (Q2_K) to near-lossless (Q8_0).

## ⚠️ Limitations

- This is a **merge**, not a model trained from scratch — behavior is a blend of its parent models and may inherit their quirks or biases.
- Lower-bit quantizations (Q2_K, Q3_K_M) trade quality for size; expect more inconsistency at extreme compression.
- Not evaluated on standard benchmarks yet — treat outputs with normal LLM caution, especially for factual or high-stakes use.

## 🙏 Credits

- Base models: [Qwen/Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct), [deepseek-ai/DeepSeek-R1-Distill-Qwen-7B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B)
- Merge tooling: [mergekit](https://github.com/arcee-ai/mergekit)
- Quantization tooling: [llama.cpp](https://github.com/ggerganov/llama.cpp)

---

<div align="center">

**If this was useful, a ⭐️ on the repo helps others find it.**

</div>