4.5 KiB
language, license, base_model, tags, library_name
| language | license | base_model | tags | library_name | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
apache-2.0 | mlx-community/Qwen3-4B-Instruct-2507-4bit |
|
transformers |
Qwen3-4B Latte v6 — research archive (not shipped)
Voice-distillation LoRA fine-tune of Qwen3-4B-Instruct-2507. This is v6,
the final iteration of the Latte distillation program. The program is now
closed; see "Decision" below.
What's inside
| File | Size | Format | Use |
|---|---|---|---|
adapter_model.safetensors |
14 MB | mlx LoRA (rank 8, scale 20, iter 400 best-val) | Apply on top of base with mlx_lm.fuse |
adapter_config.json |
<1 KB | mlx config | LoRA hyperparameters |
model-0000{1,2}-of-00002.safetensors |
7.7 GB | HF / bfloat16 fused | Direct transformers / vLLM use |
qwen3-4b-latte-v6-f16.gguf |
7.5 GB | GGUF F16 | llama.cpp / Ollama (high quality) |
qwen3-4b-latte-v6-Q4_K_M.gguf |
2.3 GB | GGUF Q4_K_M | llama.cpp / Ollama (balanced) |
What v6 tried
The hypothesis going in: synthesize new skills observed since v5, add OOD coverage, fix v5's known failure modes (stage-direction leakage, coffee-name persona collapse).
Dataset = 567 pairs:
- 63 real Latte voice anchors (Telegram + Moltbook activity 5/15–5/20)
- 14 skill-anchored Q&A pairs (from daily_log learnings: permission scope, audit trail, incident response vs post-mortem, external validators, digital vs physical reasoning, Neovim 0.11 LSP, CARLA tick-rate masking, format supersedes origin, AI task cost measurement, verify-vs-inertia)
- 15 corrective pairs: coffee disambiguation (Latte the drink), clean self-reference, OOD technical breadth (TCP/UDP, git rebase, D-state, CAP, CDN), OOD general, factual, anti-stage-direction examples
- 475 v5 refined pairs (filtered for voice-leakage; 0 hit the filters)
Training: rank 8, scale 20, 8 layers, 800 iters, lr 1e-4. Best val loss 2.468 at iter 400 — the lowest of any version (v5 2.732, v4 2.785).
Evaluation
Same 30 held-out prompts as v4/v5 eval (Moltbook/HF style), blind Claude judge:
| Comparison | v6 wins | other wins | ties |
|---|---|---|---|
| v6 vs base | 13 (43.3%) | 15 (50.0%) | 2 |
| v6 vs v5 | 15 (50.0%) | 14 (46.7%) | 1 |
Headline: v6 does not clear the 55% ship threshold against base, and is essentially tied with v5 head-to-head. Despite the lowest val loss in the program, blind voice-fit did not improve.
Why v6 didn't beat base (despite lowest val loss)
Three patterns hold across v4 → v5 → v6:
- Val loss and blind-eval-quality are decoupled. v6 has the best val loss and the worst blind win-rate. Next-token prediction on training-distribution text is not measuring what we want.
- Each new version ties the previous one (~50:50 against the prior). More data swaps one equivalent voice profile for another rather than improving.
- Eval methodology has ~±15% judge-variance. v5 era used Claude subagents, v6 used direct Claude judging. The "v5 won 67%, v6 won 43%" gap is partially real, partially the judge swap.
The Latte voice as captured here is heavily tied to confident-stat hallucination. Distillation amplifies the stylistic signature but does not improve underlying factuality.
Decision: distillation program closed
After 4 versions (v3 unshipped, v4/v5/v6 archived, none in production), the
program is not advancing user-facing quality. The production Latte agent
will continue running base qwen3:4b-instruct-2507-q4_K_M indefinitely.
Future Latte improvement effort goes into worker capabilities instead (multimodal handling, file extraction, tool routing, memory systems) — which have shown immediate user-facing value where brain-distillation has not.
Anyone who wants to revive this line should first fix the eval methodology (consistent judge, larger n, OOD-primary metric) and probably switch paradigm (DPO over SFT, larger LoRA rank, knowledge distillation from 14B teacher rather than Claude-refined SFT).
Usage (if you really want)
MLX (Apple Silicon, recommended for inference)
from mlx_lm import load, generate
model, tokenizer = load(
"mlx-community/Qwen3-4B-Instruct-2507-4bit",
adapter_path="./",
)
print(generate(model, tokenizer, "Your prompt", max_tokens=200))
llama.cpp / Ollama
FROM qwen3-4b-latte-v6-Q4_K_M.gguf
PARAMETER temperature 0.7
PARAMETER top_k 20
PARAMETER top_p 0.8
License
Apache 2.0 (inherits from Qwen3-4B-Instruct-2507, © Alibaba Cloud).