97 lines
4.9 KiB
Markdown
97 lines
4.9 KiB
Markdown
---
|
||
license: other
|
||
license_name: qwen-research
|
||
license_link: https://huggingface.co/Qwen/Qwen2.5-3B-Instruct/blob/main/LICENSE
|
||
base_model: Qwen/Qwen2.5-3B-Instruct
|
||
tags:
|
||
- merge
|
||
- ties
|
||
- model-merging
|
||
- research
|
||
- tetracta
|
||
language:
|
||
- en
|
||
pipeline_tag: text-generation
|
||
---
|
||
|
||
# Chimera-1 — two skills, one model, training-free merge (method demo)
|
||
|
||
**Chimera-1** (*chimera*: in biology, one organism carrying two genetic lineages — which is exactly what this model is) is a proof-of-concept for a simple question: *can you take two specialists and stitch
|
||
their skills into ONE model — with no training at the merge step, and no loss of general ability?*
|
||
|
||
Answer, measured: **yes, under conditions we state below.**
|
||
|
||
| model | skill A (Roman numerals) | skill B (letter ops) | Belebele-EN | perplexity | chat |
|
||
|---|---|---|---|---|---|
|
||
| Qwen2.5-3B-Instruct (base) | 76% | 45% | 50% | 68.1 | ✓ |
|
||
| specialist A | **98%** | 45% | — | — | — |
|
||
| specialist B | 80% | **75%** | — | — | — |
|
||
| **Chimera-1 (this model)** | **89%** | **71%** | **50%** | **68.1** | ✓ identical |
|
||
|
||
**The headline:** +13 and +26 points on the two target skills, and **zero measured regression** on general
|
||
ability — Belebele-English, fixed-text perplexity and chat outputs are bit-for-bit indistinguishable from the base.
|
||
The merge itself is **training-free** (TIES: task-vector trim + sign election + disjoint mean), runs on CPU in minutes,
|
||
and the result is a single 3B model with **no extra inference cost**.
|
||
|
||
## What this is — and is not
|
||
|
||
- **It is a method demo.** The two skills (Roman-numeral conversion, character-level word operations) are deliberately
|
||
simple, verifiable testbeds — chosen because the base model is measurably weak at them (headroom). This is not a
|
||
production assistant; it is evidence about *merging*.
|
||
- **The specialists were made by us** with light fine-tuning (top-layers only) — the *merge* is the training-free part.
|
||
- **The law we validated across 10 experiments:** merge synergy = **headroom × complementarity × proximity.**
|
||
If the base already solves the task, there is nothing to gain. If the parents aren't genuinely complementary,
|
||
merging dilutes. If a parent has drifted far from the shared base (heavy continued-pretraining), aggressive
|
||
merging produces mush — we show a negative control where a merged coder model drops to 0% on code.
|
||
- **What we did not manage (stated honestly):** three attempts at X-Ray-*guided* merge surgery (per-layer and
|
||
per-module weighting/refereeing from internal divergence maps) did **not** beat uniform TIES on average — one
|
||
variant preserved the concentrated skill best (95%) at the cost of the other. Blind TIES is a strong baseline.
|
||
What internal measurement *did* reliably do is **predict which pairs merge profitably before merging** (3/3 in
|
||
our runs) and expose the trade-off dial.
|
||
|
||
|
||
## Full head-to-head vs the base — 1,210 questions (22 Jul)
|
||
|
||
Same LL-based harness for both models. Sanity check: the base scores **MMLU 67.0%**, matching its publicly
|
||
reported numbers — the harness measures correctly.
|
||
|
||
|
||
|
||
| benchmark | base | Chimera-1 | Δ |
|
||
|---|---|---|---|
|
||
| MMLU (200) | 67.0% | 66.5% | −0.5 |
|
||
| ARC-Challenge (200) | 81.5% | 81.5% | 0.0 |
|
||
| HellaSwag (150) | 70.0% | 71.3% | +1.3 |
|
||
| Belebele-EN (200) | 51.0% | 50.5% | −0.5 |
|
||
| Belebele-TR (75) | 41.3% | 41.3% | 0.0 |
|
||
| Belebele-AR (75) | 50.7% | 49.3% | −1.3 |
|
||
| Belebele-ZH (75) | 50.7% | 50.7% | 0.0 |
|
||
| Belebele-RU (75) | 49.3% | 49.3% | 0.0 |
|
||
| **Roman numerals (80)** | 82.5% | **91.2%** | **+8.8** |
|
||
| **Letter ops (80)** | 37.5% | **62.5%** | **+25.0** |
|
||
|
||
Every standard benchmark sits inside the ±2pp bf16 run-noise band (max |Δ| = 1.3pp); the two implanted skills
|
||
moved +8.8 and +25.0. A capability edit with a **measured blast radius of zero** — verified on 1,050 standard
|
||
questions, not assumed from a spot check.
|
||
|
||
## Why we built it
|
||
|
||
We build an interpretability instrument (an "X-Ray" for LLMs) at [tetracta.ai](https://www.tetracta.ai/xray.html).
|
||
Model merging is the perfect stress-test for it: everyone merges blind and evals after; we wanted to know how much
|
||
of the outcome is predictable *from the weights, before the merge*. Full write-up, per-layer divergence films and
|
||
the honest failure catalogue: [tetracta.ai/research.html](https://www.tetracta.ai/research.html).
|
||
|
||
## Use
|
||
|
||
Qwen2.5 ChatML format, identical to the base model. License inherited from Qwen2.5-3B-Instruct
|
||
(**Qwen Research License — non-commercial, research use**).
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
m = AutoModelForCausalLM.from_pretrained("tetracta/Chimera-1-Qwen2.5-3B", dtype="bfloat16")
|
||
t = AutoTokenizer.from_pretrained("tetracta/Chimera-1-Qwen2.5-3B")
|
||
# try: "Convert the Roman numeral CDXLIV to a regular number." / "Spell the word 'measure' backwards."
|
||
```
|
||
|
||
*— Tetracta AI Teams · for humans, like humans. We measure; we publish the failures too.*
|