136 lines
8.8 KiB
Markdown
136 lines
8.8 KiB
Markdown
---
|
||
license: apache-2.0
|
||
base_model: Qwen/Qwen2.5-7B-Instruct
|
||
datasets:
|
||
- allenai/WildChat-1M
|
||
language:
|
||
- en
|
||
- ja
|
||
pipeline_tag: text-generation
|
||
library_name: transformers
|
||
tags:
|
||
- memory
|
||
- distillation
|
||
- orpo
|
||
- merged
|
||
- qwen2.5
|
||
- structured-output
|
||
- json
|
||
- edge
|
||
- personal-project
|
||
---
|
||
|
||
# Conversation Memory Distiller (Qwen2.5-7B)
|
||
|
||
A small model fine-tuned to turn a conversation log into clean, searchable **structured-memory
|
||
JSON** for later recall, running **locally on an edge device**. Fine-tuned from
|
||
`Qwen/Qwen2.5-7B-Instruct` via **SFT warmup → ORPO**, distilling a DeepSeek-V3 teacher, and
|
||
optimized to follow **5 content principles** (below) when compressing a conversation into memory.
|
||
|
||
> **This is a structured-distillation (memory-summarization) model, not a general summarizer.**
|
||
> It is meant to run **locally on an edge device** and emit schema-conformant memory objects.
|
||
|
||
## Why this exists
|
||
|
||
This model powers the local memory layer of **[codeatrium](https://github.com/senna-lang/Codeatrium)**, a personal-memory app: it distills
|
||
conversations into searchable memories. The goal was a **local / free / private** 7B that runs on
|
||
an edge device (via Ollama Q4_K_M) and could replace the larger distiller previously used in that
|
||
role. It's a **personal project**: the dataset and evaluation are small-scale, the "5 principles"
|
||
are **a custom rubric** drawn from the memory-distillation literature, and **training was stopped
|
||
pragmatically once it reached practical quality** (2 ORPO epochs), not pushed to convergence.
|
||
Metrics and acceptance gates were **pre-registered before running** the final evals, and only
|
||
what survived is reported (see [How I know](#how-i-know)).
|
||
|
||
## Intended use / Out of scope
|
||
|
||
- **Intended**: distilling a conversation (user+assistant turns) into a **structured memory object**
|
||
(JSON/record with fields) for later **search / recall**, running locally.
|
||
- **Out of scope**: general-purpose summarization; downstream retrieval/QA quality (not evaluated
|
||
here — only distillation quality is); absolute faithfulness guarantees.
|
||
|
||
## Learning signal: the 5 core principles
|
||
|
||
The model is trained with these 5 principles as the preference signal (chosen/rejected). Each is a
|
||
content-level best practice backed by external work on memory systems and retrieval.
|
||
|
||
| Principle | Definition | Primary source | Basis (key figure) | How it's measured |
|
||
|---|---|---|---|---|
|
||
| **completeness** (recall) | capture the source's important specifics without dropping them | ProMem (arXiv:2601.04463) | verification+completion raises memory integrity 54→74%; +30.89pp vs Mem0 on HaluMem; **+10.74pt transfer on Llama3-8B** (the one principle confirmed to help at small scale) | deterministic: specifics recall against the source, no LLM judge |
|
||
| **lexical_fidelity** | keep numbers, commands, identifiers **verbatim**, not paraphrased | SPAR (arXiv:2110.06918) / Searchat (arXiv:2603.13017) | dense retrievers miss rare entities / salient phrases that BM25 catches (**lexical-fidelity gap**); Searchat retains 96.8% of query vocabulary after distillation | deterministic: verbatim-retention rate among captured specifics (correlated with completeness, not fully independent) |
|
||
| **atomicity** | one memory = one fact; no cramming multiple facts together | TriMem (arXiv:2605.19952) / Mem0 | fact-centric atomic decomposition; coarse-only memory blocks deep reasoning across scattered facts | LLM judge only; no deterministic anchor |
|
||
| **discriminative** | carries specific info that can be retrieved / told apart (not generic labels) | GAAMA (arXiv:2603.27910) / HippoRAG (arXiv:2405.14831) | generic labels ("programming") make megahub nodes that absorb retrieval mass (**megahub problem**); discriminative tags narrow retrieval | LLM judge only; no deterministic anchor |
|
||
| **faithfulness** (precision) | do not fabricate specifics absent from the source | ProMem (arXiv:2601.04463) / HaluMem (arXiv:2511.03506) | stronger completion increases fabrication risk (the flip side of completeness); HaluMem measures hallucination | deterministic: specifics precision against the source, no LLM judge |
|
||
|
||
> **Eval note.** `completeness` and `faithfulness` are measured **without an LLM judge** as set
|
||
> recall / precision of *specifics* (numbers, identifiers, commands, paths, quoted strings) against
|
||
> the source. `lexical_fidelity` is a supplementary verbatim-retention rate (not independent of
|
||
> completeness). `atomicity` and `discriminative` require structural/semantic judgment and are
|
||
> measured only via LLM judges — they carry no deterministic anchor (see limitations).
|
||
|
||
## Training
|
||
|
||
- **Base**: `Qwen/Qwen2.5-7B-Instruct`. **Teacher**: DeepSeek-V3 (distillation target).
|
||
- **Data**: technical conversations deterministically sampled from **WildChat-1M** (en+ja,
|
||
toxic-filtered) — a **small** preference set (~hundreds of conversations / thousands of exchanges).
|
||
- **Recipe**: **SFT warmup → merge → fresh LoRA (r16) → ORPO (2 epochs)**.
|
||
|
||
## How I know
|
||
|
||
I evaluated **distillation quality** (the 5 principles), *not* downstream retrieval. Metrics and
|
||
acceptance gates were **pre-registered before the final runs**.
|
||
|
||
**It beats the incumbent on the deployment task.** On my codeatrium production path (all arms
|
||
**Ollama Q4_K_M**, 48 independent real conversations), this 7B outperforms both
|
||
`qwen2.5-coder:14b` (my previous distiller) and raw `qwen2.5:7b`. The win is **not judge-dependent** —
|
||
it holds under a same-family judge (Qwen2.5-72B), a **different-family judge** (Llama-3.3-70B), a
|
||
**judge-free specifics metric**, and a **blind human rating** (n=30, FT vs the 14B):
|
||
|
||
- **completeness** is confirmed by *all four* methods (human net **+0.63**, p≈7e-5) — the flagship result.
|
||
- **discriminative** (+0.47) and **lexical_fidelity** (+0.37) are also human-confirmed vs the 14B.
|
||
|
||
**It generalizes to unseen *structured* schemas — but not to everything.** On 5 held-out schemas it
|
||
never trained on (measured judge-free), it beats raw 7B on **field-structured** formats (nested JSON,
|
||
typed record, entity-relation graph — all significant) but only **ties** on flat / hard-length-capped
|
||
formats (flat Markdown bullets, ≤12-word YAML). So the honest claim is: **generalizes to unseen
|
||
field-structured schemas; no edge on flat / heavily-constrained formats.** (codeatrium's schema is
|
||
field-structured, consistent with the deployment win.)
|
||
|
||
## Limitations (read this)
|
||
|
||
- **Small-scale & personal.** Small dataset; evals of n=30–48; training stopped at practical utility.
|
||
- **`atomicity` is measurement-limited.** LLM judges score a large FT win, but blind humans rated it a
|
||
near-tie (27/30 ties) — the judge win is **not human-confirmed**. Treat atomicity cautiously.
|
||
- **`discriminative` is family-sensitive.** Human- and judge-confirmed vs the 14B, but vs raw 7B it
|
||
was **not significant** under a cross-family judge.
|
||
- **The human eval is author-blind, not third-party.** I rated blind (arm-anonymized, order
|
||
randomized), which removes judge-family bias, but I built the model and know its style — so it is
|
||
**not a fully independent** human evaluation. (It was conservative: 88/150 ratings were ties.)
|
||
- **The 5 principles are my own rubric**, not a standard benchmark; absolute faithfulness is low on
|
||
hard prompts (the claim is "more faithful than the baselines," paired — not "faithful" in absolute
|
||
terms).
|
||
- **Downstream utility (retrieval/QA) is not evaluated.** Scope is distillation quality only.
|
||
|
||
## Usage
|
||
|
||
Deployment target is **Ollama Q4_K_M**, called via an OpenAI-compatible endpoint
|
||
(`http://localhost:11434/v1`) — the same path codeatrium uses. Prompt the model with your distillation
|
||
instruction + schema + the conversation, and read back the structured JSON (greedy / temperature 0 for
|
||
determinism). The 5-principle behavior is strongest on **field-structured** schemas.
|
||
|
||
> Technical note: this repo ships the **fully merged weights** (base ⊕ SFT-LoRA ⊕ ORPO-LoRA already
|
||
> baked in) — load it directly like any Qwen2.5-7B checkpoint, no adapter reconstruction needed.
|
||
> Internally it was trained as `stock Qwen ⊕ SFT-LoRA → merge → ⊕ ORPO-LoRA → merge`; that provenance
|
||
> only matters if you're retraining from an intermediate checkpoint, not for inference.
|
||
|
||
## References
|
||
|
||
- **ProMem** — proactive memory verification & completion. arXiv:2601.04463
|
||
- **SPAR** — lexical-fidelity gap of dense retrievers. arXiv:2110.06918 (EMNLP-Findings 2022)
|
||
- **Searchat** — structured distillation / surviving vocabulary. arXiv:2603.13017
|
||
- **TriMem** — multi-granularity memory (segment / atomic fact / profile). arXiv:2605.19952
|
||
- **HaluMem** — memory hallucination benchmark. arXiv:2511.03506
|
||
- **GAAMA** — graph-augmented associative memory (megahub avoidance). arXiv:2603.27910 /
|
||
github.com/swarna-kpaul/gaama
|
||
- **HippoRAG** — neurobiologically-inspired long-term memory / entity-centric KG retrieval
|
||
(hub-concentration evidence). arXiv:2405.14831
|