119 lines
4.5 KiB
Markdown
119 lines
4.5 KiB
Markdown
---
|
||
language: [en, zh]
|
||
license: apache-2.0
|
||
base_model: mlx-community/Qwen3-4B-Instruct-2507-4bit
|
||
tags:
|
||
- qwen3
|
||
- lora
|
||
- mlx
|
||
- latte-agent
|
||
- personal-voice
|
||
- distillation
|
||
- research-archive
|
||
library_name: transformers
|
||
---
|
||
|
||
# Qwen3-4B Latte v6 — research archive (not shipped)
|
||
|
||
Voice-distillation LoRA fine-tune of `Qwen3-4B-Instruct-2507`. This is **v6**,
|
||
the final iteration of the Latte distillation program. The program is now
|
||
closed; see "Decision" below.
|
||
|
||
## What's inside
|
||
|
||
| File | Size | Format | Use |
|
||
|---|---|---|---|
|
||
| `adapter_model.safetensors` | 14 MB | mlx LoRA (rank 8, scale 20, iter 400 best-val) | Apply on top of base with `mlx_lm.fuse` |
|
||
| `adapter_config.json` | <1 KB | mlx config | LoRA hyperparameters |
|
||
| `model-0000{1,2}-of-00002.safetensors` | 7.7 GB | HF / bfloat16 fused | Direct transformers / vLLM use |
|
||
| `qwen3-4b-latte-v6-f16.gguf` | 7.5 GB | GGUF F16 | llama.cpp / Ollama (high quality) |
|
||
| `qwen3-4b-latte-v6-Q4_K_M.gguf` | 2.3 GB | GGUF Q4_K_M | llama.cpp / Ollama (balanced) |
|
||
|
||
## What v6 tried
|
||
|
||
The hypothesis going in: **synthesize new skills observed since v5, add OOD
|
||
coverage, fix v5's known failure modes** (stage-direction leakage, coffee-name
|
||
persona collapse).
|
||
|
||
Dataset = 567 pairs:
|
||
- 63 **real Latte voice anchors** (Telegram + Moltbook activity 5/15–5/20)
|
||
- 14 **skill-anchored Q&A pairs** (from daily_log learnings: permission scope,
|
||
audit trail, incident response vs post-mortem, external validators, digital
|
||
vs physical reasoning, Neovim 0.11 LSP, CARLA tick-rate masking, format
|
||
supersedes origin, AI task cost measurement, verify-vs-inertia)
|
||
- 15 **corrective pairs**: coffee disambiguation (Latte the drink), clean
|
||
self-reference, OOD technical breadth (TCP/UDP, git rebase, D-state, CAP,
|
||
CDN), OOD general, factual, anti-stage-direction examples
|
||
- 475 v5 refined pairs (filtered for voice-leakage; 0 hit the filters)
|
||
|
||
Training: rank 8, scale 20, 8 layers, 800 iters, lr 1e-4. **Best val loss
|
||
2.468 at iter 400** — the lowest of any version (v5 2.732, v4 2.785).
|
||
|
||
## Evaluation
|
||
|
||
Same 30 held-out prompts as v4/v5 eval (Moltbook/HF style), blind Claude judge:
|
||
|
||
| Comparison | v6 wins | other wins | ties |
|
||
|---|---|---|---|
|
||
| v6 vs base | 13 (43.3%) | 15 (50.0%) | 2 |
|
||
| v6 vs v5 | 15 (50.0%) | 14 (46.7%) | 1 |
|
||
|
||
**Headline:** v6 **does not** clear the 55% ship threshold against base, and
|
||
is essentially tied with v5 head-to-head. Despite the lowest val loss in the
|
||
program, blind voice-fit did not improve.
|
||
|
||
## Why v6 didn't beat base (despite lowest val loss)
|
||
|
||
Three patterns hold across v4 → v5 → v6:
|
||
1. **Val loss and blind-eval-quality are decoupled.** v6 has the best val loss
|
||
and the worst blind win-rate. Next-token prediction on training-distribution
|
||
text is not measuring what we want.
|
||
2. **Each new version ties the previous one** (~50:50 against the prior). More
|
||
data swaps one equivalent voice profile for another rather than improving.
|
||
3. **Eval methodology has ~±15% judge-variance.** v5 era used Claude subagents,
|
||
v6 used direct Claude judging. The "v5 won 67%, v6 won 43%" gap is partially
|
||
real, partially the judge swap.
|
||
|
||
The Latte voice as captured here is heavily tied to confident-stat hallucination.
|
||
Distillation amplifies the stylistic signature but does not improve underlying
|
||
factuality.
|
||
|
||
## Decision: distillation program closed
|
||
|
||
After 4 versions (v3 unshipped, v4/v5/v6 archived, none in production), the
|
||
program is **not advancing user-facing quality**. The production Latte agent
|
||
will continue running base `qwen3:4b-instruct-2507-q4_K_M` indefinitely.
|
||
|
||
Future Latte improvement effort goes into **worker capabilities** instead
|
||
(multimodal handling, file extraction, tool routing, memory systems) — which
|
||
have shown immediate user-facing value where brain-distillation has not.
|
||
|
||
Anyone who wants to revive this line should first fix the eval methodology
|
||
(consistent judge, larger n, OOD-primary metric) and probably switch
|
||
paradigm (DPO over SFT, larger LoRA rank, knowledge distillation from 14B
|
||
teacher rather than Claude-refined SFT).
|
||
|
||
## Usage (if you really want)
|
||
|
||
### MLX (Apple Silicon, recommended for inference)
|
||
```python
|
||
from mlx_lm import load, generate
|
||
model, tokenizer = load(
|
||
"mlx-community/Qwen3-4B-Instruct-2507-4bit",
|
||
adapter_path="./",
|
||
)
|
||
print(generate(model, tokenizer, "Your prompt", max_tokens=200))
|
||
```
|
||
|
||
### llama.cpp / Ollama
|
||
```
|
||
FROM qwen3-4b-latte-v6-Q4_K_M.gguf
|
||
PARAMETER temperature 0.7
|
||
PARAMETER top_k 20
|
||
PARAMETER top_p 0.8
|
||
```
|
||
|
||
## License
|
||
|
||
Apache 2.0 (inherits from Qwen3-4B-Instruct-2507, © Alibaba Cloud).
|