57 lines
1.7 KiB
Markdown
57 lines
1.7 KiB
Markdown
|
---
|
|||
|
|
license: apache-2.0
|
|||
|
|
base_model: build-small-hackathon/deku
|
|||
|
|
tags:
|
|||
|
|
- gguf
|
|||
|
|
- llama.cpp
|
|||
|
|
- knowledge-distillation
|
|||
|
|
- qwen2
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# Deku — GGUF (llama.cpp)
|
|||
|
|
|
|||
|
|
GGUF builds of [**build-small-hackathon/deku**](https://huggingface.co/build-small-hackathon/deku),
|
|||
|
|
the One for All student: a Qwen2.5-0.5B distilled from 6 teachers via gated CKA
|
|||
|
|
geometry distillation. The LoRA adapter is merged into the base, then converted
|
|||
|
|
with `llama.cpp`'s `convert_hf_to_gguf.py`.
|
|||
|
|
|
|||
|
|
## Files
|
|||
|
|
|
|||
|
|
| File | Size | Use |
|
|||
|
|
|------|------|-----|
|
|||
|
|
| `deku-q8_0.gguf` | ~531 MB | what the Space serves — near-lossless, CPU-friendly |
|
|||
|
|
| `deku-f16.gguf` | ~994 MB | archival full-precision build |
|
|||
|
|
| `gating.npz` | ~22 KB | the teacher-gating head as numpy (`weight` 6×896, `bias` 6) |
|
|||
|
|
|
|||
|
|
## Run
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
llama-cli -m deku-q8_0.gguf -p "Explain gradient descent in one sentence."
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
from llama_cpp import Llama
|
|||
|
|
llm = Llama(model_path="deku-q8_0.gguf", n_ctx=2048)
|
|||
|
|
print(llm.create_chat_completion(
|
|||
|
|
messages=[{"role": "user", "content": "Why is the sky blue?"}]
|
|||
|
|
)["choices"][0]["message"]["content"])
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## Teacher gating without torch
|
|||
|
|
|
|||
|
|
`gating.npz` lets you reproduce the live "teacher influence" meters from the
|
|||
|
|
[Space](https://huggingface.co/spaces/build-small-hackathon/one-for-all) using
|
|||
|
|
only numpy on a mean-pooled embedding from `llama.cpp`:
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
import numpy as np
|
|||
|
|
g = np.load("gating.npz") # g["weight"] (6, 896), g["bias"] (6,)
|
|||
|
|
def gate(emb): # emb: 896-dim pooled embedding
|
|||
|
|
z = g["weight"] @ emb + g["bias"]
|
|||
|
|
e = np.exp(z - z.max())
|
|||
|
|
return e / e.sum() # softmax over the 6 teachers
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Teacher order: `qwen, smollm, phi, gemma, minicpm, nemotron`.
|