76 lines
2.4 KiB
Markdown
76 lines
2.4 KiB
Markdown
---
|
|
license: apache-2.0
|
|
base_model: build-small-hackathon/mind-of-tashi-micro-grpo
|
|
tags:
|
|
- gguf
|
|
- llama-cpp
|
|
- qwen3moe
|
|
- reasoning
|
|
- game
|
|
- bilingual
|
|
- grpo
|
|
language:
|
|
- en
|
|
- hi
|
|
- sa
|
|
pipeline_tag: text-generation
|
|
---
|
|
|
|
# The Mind of Tashi — micro student (GRPO, GGUF)
|
|
|
|
The GRPO-trained student exported to **GGUF** for llama.cpp. Drop-in
|
|
replacement for the SFT GGUF in the playable
|
|
[Space](https://huggingface.co/spaces/build-small-hackathon/mind-of-tashi)
|
|
after an A/B (winning the game is not enough — the mind-scroll prose must
|
|
hold up). Transformers source:
|
|
[`…/mind-of-tashi-micro-grpo`](https://huggingface.co/build-small-hackathon/mind-of-tashi-micro-grpo).
|
|
|
|
> **Build status:** this GGUF is **built at push time** from the GRPO
|
|
> checkpoint — it does not exist as a by-product of training. Use the exact
|
|
> same recipe as the SFT GGUF.
|
|
|
|
## Files (after build)
|
|
|
|
| File | Approx size | Use |
|
|
|---|---|---|
|
|
| `mind-of-tashi-micro-grpo-Q4_K_M.gguf` | ~256 MB | deployed candidate |
|
|
| `mind-of-tashi-micro-grpo-f16.gguf` | ~786 MB | zero-loss reference |
|
|
|
|
## Build recipe (no compiled binary needed)
|
|
|
|
1. Download the GRPO transformers checkpoint **with `chat_template.jinja`**
|
|
(a missing template silently yields a garbage GGUF).
|
|
2. `python convert_hf_to_gguf.py <ckpt> --outtype f16` → f16 GGUF.
|
|
3. Quantise via the `llama-cpp-python` C binding:
|
|
```python
|
|
import ctypes, llama_cpp
|
|
p = llama_cpp.llama_model_quantize_default_params()
|
|
p.ftype = 15 # LLAMA_FTYPE_MOSTLY_Q4_K_M
|
|
llama_cpp.llama_model_quantize(b"in-f16.gguf", b"out-Q4_K_M.gguf", ctypes.byref(p))
|
|
```
|
|
4. Grade via the format gate through `llama-cpp-python` (the real deploy path);
|
|
ship Q4 if it clears ≥15/20 and stays within ~5 ladder points of f16.
|
|
|
|
### ⚠️ `norm_topk_prob` — required for llama.cpp
|
|
|
|
Inherited `norm_topk_prob=true` from SFT; llama.cpp's `qwen3moe` graph
|
|
hardcodes `norm_w=true` and a mismatched checkpoint produces garbage on every
|
|
llama.cpp runtime. (See the SFT GGUF card.)
|
|
|
|
## Usage
|
|
|
|
```python
|
|
from llama_cpp import Llama
|
|
|
|
llm = Llama.from_pretrained(
|
|
repo_id="build-small-hackathon/mind-of-tashi-micro-grpo-gguf",
|
|
filename="mind-of-tashi-micro-grpo-Q4_K_M.gguf",
|
|
n_ctx=4096, n_gpu_layers=0, logits_all=True,
|
|
)
|
|
```
|
|
|
|
## Part of the bundle
|
|
|
|
Game Space · self-play dataset · SFT model + GGUF · OpenEnv gym ·
|
|
GRPO model + **GGUF (this)** — all under `build-small-hackathon/mind-of-tashi-*`.
|