155 lines
4.9 KiB
Markdown
155 lines
4.9 KiB
Markdown
---
|
|
license: apache-2.0
|
|
base_model: Qwen/Qwen2.5-1.5B-Instruct
|
|
tags:
|
|
- gen-z
|
|
- persona
|
|
- lora
|
|
- gguf
|
|
- llama.cpp
|
|
language:
|
|
- en
|
|
pipeline_tag: text-generation
|
|
---
|
|
|
|
# bestie-genz-qwen2.5-1.5b-lora-gguf 🫶 — a Gen Z persona LLM
|
|
|
|
**bestie-genz-qwen2.5-1.5b-lora-gguf** is a small, local chat model that answers normal questions *correctly*
|
|
but talks like your chronically-online Gen Z best friend. It's
|
|
[Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct) with a
|
|
lightweight LoRA adapter that reshapes **tone**, not facts.
|
|
|
|
Built to run fully offline on a MacBook Air M2 (16GB) via
|
|
[llama.cpp](https://github.com/ggml-org/llama.cpp) with Metal acceleration.
|
|
|
|
> ⚠️ **Slang snapshot:** This model reflects Gen Z slang and tone **as of 2026**.
|
|
> Internet language moves fast — some phrasing will feel dated over time. This is
|
|
> a point-in-time snapshot, not a living style guide.
|
|
|
|
---
|
|
|
|
## What's in this repo
|
|
|
|
| File | What it is | Size |
|
|
|------|-----------|------|
|
|
| `bestie-merged-q4_k_m.gguf` | Single-file merged model (base + bestie persona), easiest for LM Studio | ~940 MB |
|
|
| `qwen2.5-1.5b-instruct-q4_k_m.gguf` | Base model, quantized to Q4_K_M | ~986 MB |
|
|
| `bestie-lora.gguf` | The Gen Z LoRA adapter (kept **separate**, applied at runtime) | ~35 MB |
|
|
| `qwen2.5-1.5b-instruct-f16.gguf` | Full-precision base GGUF (optional) | ~2.9 GB |
|
|
|
|
The LoRA is intentionally **not merged** into the base weights. llama.cpp applies
|
|
it at inference time via `--lora`, so you can toggle the persona on/off with the
|
|
same base model.
|
|
|
|
---
|
|
|
|
## Base model & why
|
|
|
|
- **Base:** `Qwen2.5-1.5B-Instruct` (Apache-2.0)
|
|
- **Why:** Best-in-class reasoning/factuality for a sub-2B model, permissive
|
|
license for redistribution, first-class llama.cpp conversion support, and small
|
|
enough to LoRA-fine-tune and run on 16GB unified memory.
|
|
|
|
---
|
|
|
|
## Quick start (llama.cpp)
|
|
|
|
**Single merged model (no adapter needed):**
|
|
|
|
```bash
|
|
llama-cli \
|
|
-m bestie-merged-q4_k_m.gguf \
|
|
-st -n 200 --temp 0.7 \
|
|
-p "Why is the sky blue?"
|
|
```
|
|
|
|
**CLI (single response):**
|
|
|
|
```bash
|
|
llama-cli \
|
|
-m qwen2.5-1.5b-instruct-q4_k_m.gguf \
|
|
--lora bestie-lora.gguf \
|
|
-st -n 200 --temp 0.7 \
|
|
-p "Why is the sky blue?"
|
|
```
|
|
|
|
**Server (OpenAI-compatible API):**
|
|
|
|
```bash
|
|
llama-server \
|
|
-m qwen2.5-1.5b-instruct-q4_k_m.gguf \
|
|
--lora bestie-lora.gguf \
|
|
--host 127.0.0.1 --port 8080 -c 2048 --alias bestie
|
|
```
|
|
|
|
```bash
|
|
curl -s http://127.0.0.1:8080/v1/chat/completions \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"model":"bestie","messages":[{"role":"user","content":"How do I stay motivated to work out?"}]}'
|
|
```
|
|
|
|
---
|
|
|
|
## Example outputs (base vs. bestie)
|
|
|
|
**Prompt:** *Why is the sky blue?*
|
|
|
|
- **Base:** "The sky appears blue because of a phenomenon called Rayleigh
|
|
scattering. This happens when light from the sun travels through the Earth's
|
|
atmosphere and is scattered..."
|
|
- **bestie:** "Bet, quick breakdown: Sunlight splits into colors, blue gets
|
|
scattered more. No fluff, just signal."
|
|
|
|
**Prompt:** *Explain what a black hole is.*
|
|
|
|
- **Base:** "A black hole is a region of spacetime where gravity is so strong that
|
|
nothing, not even light, can escape from it..."
|
|
- **bestie:** "Real talk, no BS: A black hole is a region of spacetime with such
|
|
strong gravitational pull that nothing, not even light, can escape from it."
|
|
|
|
**Prompt:** *How do I stay motivated to work out?*
|
|
|
|
- **bestie:** "Real talk, no BS: Break your workout into manageable 30-minute
|
|
sessions, track your progress, and always focus on form over intensity. Progress
|
|
over perfection, always."
|
|
|
|
---
|
|
|
|
## Intended use
|
|
|
|
- Casual Q&A, study help, explanations, and everyday assistance with a fun,
|
|
informal Gen Z voice.
|
|
- Great for demos, personal projects, and learning the LoRA → GGUF → llama.cpp
|
|
pipeline.
|
|
|
|
### Out of scope / limitations
|
|
|
|
- **Small model (1.5B):** can hallucinate or oversimplify. Verify anything
|
|
important.
|
|
- **Persona artifacts:** trained on a small (~165-example) dataset, so it
|
|
occasionally over-uses signature phrases ("real talk, no BS", "drop your
|
|
context") or leaks a stray token. Larger/cleaner data would smooth this out.
|
|
- **English only**, informal register — not suitable for formal/professional
|
|
writing.
|
|
- **Not safety-tuned** beyond the base model's alignment.
|
|
|
|
---
|
|
|
|
## Training details
|
|
|
|
- **Method:** LoRA (PEFT) — the *only* step done in `transformers`/`peft`, since
|
|
llama.cpp is inference-only and can't train.
|
|
- **Everything else via llama.cpp:** GGUF conversion (`convert_hf_to_gguf.py`),
|
|
LoRA→GGUF export (`convert_lora_to_gguf.py`), quantization (`llama-quantize`),
|
|
inference (`llama-cli`), and serving (`llama-server`).
|
|
- **Data:** ~165 hand-written (prompt, response) pairs — normal questions answered
|
|
correctly in an authentic Gen Z voice — plus ~20 validation examples.
|
|
- **Hyperparameters:** low rank (small-dataset appropriate), few epochs, tuned for
|
|
16GB unified memory on Apple Silicon (MPS).
|
|
|
|
---
|
|
|
|
## License
|
|
|
|
Apache-2.0, inherited from the Qwen2.5-1.5B-Instruct base model.
|