124 lines
4.2 KiB
Markdown
124 lines
4.2 KiB
Markdown
---
|
|
language: [en, zh]
|
|
license: apache-2.0
|
|
base_model: mlx-community/Qwen3-4B-Instruct-2507-4bit
|
|
tags:
|
|
- qwen3
|
|
- lora
|
|
- mlx
|
|
- latte-agent
|
|
- personal-voice
|
|
- distillation
|
|
library_name: transformers
|
|
---
|
|
|
|
# Qwen3-4B Latte v5
|
|
|
|
Voice-distillation LoRA fine-tune of `Qwen3-4B-Instruct-2507`, targeting the
|
|
private "Latte" agent persona: warm-direct, technical, takes a stance, concrete
|
|
numbers, bilingual EN/ZH, no template openers.
|
|
|
|
**This is an archival/experimental release.** It is **not** the production brain
|
|
for the live Latte agent — see eval caveats below.
|
|
|
|
## What's inside
|
|
|
|
| File | Size | Format | Use |
|
|
|---|---|---|---|
|
|
| `adapter_model.safetensors` | 14 MB | mlx LoRA (rank 8, scale 20) | Apply on top of base with `mlx_lm.fuse` |
|
|
| `adapter_config.json` | <1 KB | mlx config | LoRA hyperparameters |
|
|
| `model-0000{1,2}-of-00002.safetensors` | 8 GB | HF / bfloat16 fused | Direct transformers / vLLM use |
|
|
| `qwen3-4b-latte-v5-f16.gguf` | 7.5 GB | GGUF F16 | llama.cpp / Ollama (high quality) |
|
|
| `qwen3-4b-latte-v5-Q4_K_M.gguf` | 2.3 GB | GGUF Q4_K_M | llama.cpp / Ollama (balanced) |
|
|
|
|
## Training
|
|
|
|
- Base: `mlx-community/Qwen3-4B-Instruct-2507-4bit` (4-bit MLX)
|
|
- Method: LoRA via `mlx_lm.lora`
|
|
- LoRA: rank 8, scale 20.0, 8 layers, dropout 0
|
|
- Optimizer: Adam, lr 1e-4, batch 1, grad accum 8, grad checkpoint on
|
|
- Iters: 800 trained, **best checkpoint = iter 450** (val loss 2.732)
|
|
- Max seq: 1536, mask_prompt: true, seed: 42
|
|
- Dataset: 475 curated (instruction, response) pairs across 7 categories:
|
|
Moltbook-style comment, HF discussion reply, technical analysis (ZH),
|
|
code review snippet, persona Q&A, peer-event reply, real-time observation.
|
|
Anchored against 356 raw Latte-voice messages.
|
|
|
|
## Evaluation
|
|
|
|
30 held-out (prompt, response) pairs per pairing. Each response pair shown
|
|
blind to a Claude judge (positions randomized, model identity stripped).
|
|
|
|
| Comparison | v5 wins | base/v4 wins | ties | mean score (1-5) |
|
|
|---|---|---|---|---|
|
|
| v5 vs base | **20 (66.7%)** | 8 (26.7%) | 2 (6.7%) | v5 3.20 / base 2.93 |
|
|
| v4 vs base | 22 (73.3%) | 8 (26.7%) | 0 | v4 3.13 / base 2.70 |
|
|
| v5 vs v4 | 14 (46.7%) | 15 (50.0%) | 1 (3.3%) | v5 3.00 / v4 2.97 |
|
|
|
|
**Headline:** v5 clearly beats the un-tuned base on in-distribution
|
|
prompts (the 7 trained categories), passing the 55% ship threshold.
|
|
|
|
**Caveat 1:** v5 vs v4 is statistically a tie. Lower val loss (2.732 vs
|
|
2.785) did not produce a perceptible quality gain in blind eval. The
|
|
additional curation effort and training steps produced marginal returns.
|
|
|
|
**Caveat 2 — why this isn't production:** Out-of-distribution smoke
|
|
testing (prompts unlike the 7 training categories) shows v5 is **tied
|
|
or slightly worse than base**:
|
|
- Stage-direction leakage: v5 occasionally prefixes responses with
|
|
`"(soft, soothing Latte voice)"` — an artifact of training data that
|
|
characterized Latte's voice.
|
|
- Occasional factual regressions (e.g., confusing latte and latte
|
|
macchiato in a generic coffee Q&A).
|
|
- Reduced robustness on prompts that pull the "Latte" token toward
|
|
unrelated semantic neighborhoods (the literal coffee drink).
|
|
|
|
The 66.7% in-distribution win does not justify replacing a battle-tested
|
|
general-purpose base in production. Use this checkpoint for tasks closely
|
|
matching the 7 training categories.
|
|
|
|
## Usage
|
|
|
|
### MLX (Apple Silicon, recommended for inference)
|
|
```python
|
|
from mlx_lm import load, generate
|
|
|
|
model, tokenizer = load(
|
|
"mlx-community/Qwen3-4B-Instruct-2507-4bit",
|
|
adapter_path="./", # this repo
|
|
)
|
|
print(generate(model, tokenizer, "Your prompt", max_tokens=200))
|
|
```
|
|
|
|
### llama.cpp / Ollama
|
|
```
|
|
# Modelfile
|
|
FROM qwen3-4b-latte-v5-Q4_K_M.gguf
|
|
PARAMETER temperature 0.7
|
|
PARAMETER top_k 20
|
|
PARAMETER top_p 0.8
|
|
```
|
|
```
|
|
ollama create latte:v5 -f Modelfile
|
|
ollama run latte:v5
|
|
```
|
|
|
|
### Transformers (any platform)
|
|
```python
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained("latte-agent/qwen3-4b-latte-v5")
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
"latte-agent/qwen3-4b-latte-v5", torch_dtype="bfloat16"
|
|
)
|
|
```
|
|
|
|
## License
|
|
|
|
Inherits Apache 2.0 from base (Qwen3-4B-Instruct-2507, © Alibaba Cloud).
|
|
|
|
## Citation
|
|
|
|
If you reference this work, please cite the base model. This adapter has no
|
|
formal publication.
|