136 lines
8.8 KiB
Markdown
136 lines
8.8 KiB
Markdown
|
---
|
|||
|
|
license: apache-2.0
|
|||
|
|
base_model: Qwen/Qwen2.5-7B-Instruct
|
|||
|
|
datasets:
|
|||
|
|
- allenai/WildChat-1M
|
|||
|
|
language:
|
|||
|
|
- en
|
|||
|
|
- ja
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
library_name: transformers
|
|||
|
|
tags:
|
|||
|
|
- memory
|
|||
|
|
- distillation
|
|||
|
|
- orpo
|
|||
|
|
- merged
|
|||
|
|
- qwen2.5
|
|||
|
|
- structured-output
|
|||
|
|
- json
|
|||
|
|
- edge
|
|||
|
|
- personal-project
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# Conversation Memory Distiller (Qwen2.5-7B)
|
|||
|
|
|
|||
|
|
A small model fine-tuned to turn a conversation log into clean, searchable **structured-memory
|
|||
|
|
JSON** for later recall, running **locally on an edge device**. Fine-tuned from
|
|||
|
|
`Qwen/Qwen2.5-7B-Instruct` via **SFT warmup → ORPO**, distilling a DeepSeek-V3 teacher, and
|
|||
|
|
optimized to follow **5 content principles** (below) when compressing a conversation into memory.
|
|||
|
|
|
|||
|
|
> **This is a structured-distillation (memory-summarization) model, not a general summarizer.**
|
|||
|
|
> It is meant to run **locally on an edge device** and emit schema-conformant memory objects.
|
|||
|
|
|
|||
|
|
## Why this exists
|
|||
|
|
|
|||
|
|
This model powers the local memory layer of **[codeatrium](https://github.com/senna-lang/Codeatrium)**, a personal-memory app: it distills
|
|||
|
|
conversations into searchable memories. The goal was a **local / free / private** 7B that runs on
|
|||
|
|
an edge device (via Ollama Q4_K_M) and could replace the larger distiller previously used in that
|
|||
|
|
role. It's a **personal project**: the dataset and evaluation are small-scale, the "5 principles"
|
|||
|
|
are **a custom rubric** drawn from the memory-distillation literature, and **training was stopped
|
|||
|
|
pragmatically once it reached practical quality** (2 ORPO epochs), not pushed to convergence.
|
|||
|
|
Metrics and acceptance gates were **pre-registered before running** the final evals, and only
|
|||
|
|
what survived is reported (see [How I know](#how-i-know)).
|
|||
|
|
|
|||
|
|
## Intended use / Out of scope
|
|||
|
|
|
|||
|
|
- **Intended**: distilling a conversation (user+assistant turns) into a **structured memory object**
|
|||
|
|
(JSON/record with fields) for later **search / recall**, running locally.
|
|||
|
|
- **Out of scope**: general-purpose summarization; downstream retrieval/QA quality (not evaluated
|
|||
|
|
here — only distillation quality is); absolute faithfulness guarantees.
|
|||
|
|
|
|||
|
|
## Learning signal: the 5 core principles
|
|||
|
|
|
|||
|
|
The model is trained with these 5 principles as the preference signal (chosen/rejected). Each is a
|
|||
|
|
content-level best practice backed by external work on memory systems and retrieval.
|
|||
|
|
|
|||
|
|
| Principle | Definition | Primary source | Basis (key figure) | How it's measured |
|
|||
|
|
|---|---|---|---|---|
|
|||
|
|
| **completeness** (recall) | capture the source's important specifics without dropping them | ProMem (arXiv:2601.04463) | verification+completion raises memory integrity 54→74%; +30.89pp vs Mem0 on HaluMem; **+10.74pt transfer on Llama3-8B** (the one principle confirmed to help at small scale) | deterministic: specifics recall against the source, no LLM judge |
|
|||
|
|
| **lexical_fidelity** | keep numbers, commands, identifiers **verbatim**, not paraphrased | SPAR (arXiv:2110.06918) / Searchat (arXiv:2603.13017) | dense retrievers miss rare entities / salient phrases that BM25 catches (**lexical-fidelity gap**); Searchat retains 96.8% of query vocabulary after distillation | deterministic: verbatim-retention rate among captured specifics (correlated with completeness, not fully independent) |
|
|||
|
|
| **atomicity** | one memory = one fact; no cramming multiple facts together | TriMem (arXiv:2605.19952) / Mem0 | fact-centric atomic decomposition; coarse-only memory blocks deep reasoning across scattered facts | LLM judge only; no deterministic anchor |
|
|||
|
|
| **discriminative** | carries specific info that can be retrieved / told apart (not generic labels) | GAAMA (arXiv:2603.27910) / HippoRAG (arXiv:2405.14831) | generic labels ("programming") make megahub nodes that absorb retrieval mass (**megahub problem**); discriminative tags narrow retrieval | LLM judge only; no deterministic anchor |
|
|||
|
|
| **faithfulness** (precision) | do not fabricate specifics absent from the source | ProMem (arXiv:2601.04463) / HaluMem (arXiv:2511.03506) | stronger completion increases fabrication risk (the flip side of completeness); HaluMem measures hallucination | deterministic: specifics precision against the source, no LLM judge |
|
|||
|
|
|
|||
|
|
> **Eval note.** `completeness` and `faithfulness` are measured **without an LLM judge** as set
|
|||
|
|
> recall / precision of *specifics* (numbers, identifiers, commands, paths, quoted strings) against
|
|||
|
|
> the source. `lexical_fidelity` is a supplementary verbatim-retention rate (not independent of
|
|||
|
|
> completeness). `atomicity` and `discriminative` require structural/semantic judgment and are
|
|||
|
|
> measured only via LLM judges — they carry no deterministic anchor (see limitations).
|
|||
|
|
|
|||
|
|
## Training
|
|||
|
|
|
|||
|
|
- **Base**: `Qwen/Qwen2.5-7B-Instruct`. **Teacher**: DeepSeek-V3 (distillation target).
|
|||
|
|
- **Data**: technical conversations deterministically sampled from **WildChat-1M** (en+ja,
|
|||
|
|
toxic-filtered) — a **small** preference set (~hundreds of conversations / thousands of exchanges).
|
|||
|
|
- **Recipe**: **SFT warmup → merge → fresh LoRA (r16) → ORPO (2 epochs)**.
|
|||
|
|
|
|||
|
|
## How I know
|
|||
|
|
|
|||
|
|
I evaluated **distillation quality** (the 5 principles), *not* downstream retrieval. Metrics and
|
|||
|
|
acceptance gates were **pre-registered before the final runs**.
|
|||
|
|
|
|||
|
|
**It beats the incumbent on the deployment task.** On my codeatrium production path (all arms
|
|||
|
|
**Ollama Q4_K_M**, 48 independent real conversations), this 7B outperforms both
|
|||
|
|
`qwen2.5-coder:14b` (my previous distiller) and raw `qwen2.5:7b`. The win is **not judge-dependent** —
|
|||
|
|
it holds under a same-family judge (Qwen2.5-72B), a **different-family judge** (Llama-3.3-70B), a
|
|||
|
|
**judge-free specifics metric**, and a **blind human rating** (n=30, FT vs the 14B):
|
|||
|
|
|
|||
|
|
- **completeness** is confirmed by *all four* methods (human net **+0.63**, p≈7e-5) — the flagship result.
|
|||
|
|
- **discriminative** (+0.47) and **lexical_fidelity** (+0.37) are also human-confirmed vs the 14B.
|
|||
|
|
|
|||
|
|
**It generalizes to unseen *structured* schemas — but not to everything.** On 5 held-out schemas it
|
|||
|
|
never trained on (measured judge-free), it beats raw 7B on **field-structured** formats (nested JSON,
|
|||
|
|
typed record, entity-relation graph — all significant) but only **ties** on flat / hard-length-capped
|
|||
|
|
formats (flat Markdown bullets, ≤12-word YAML). So the honest claim is: **generalizes to unseen
|
|||
|
|
field-structured schemas; no edge on flat / heavily-constrained formats.** (codeatrium's schema is
|
|||
|
|
field-structured, consistent with the deployment win.)
|
|||
|
|
|
|||
|
|
## Limitations (read this)
|
|||
|
|
|
|||
|
|
- **Small-scale & personal.** Small dataset; evals of n=30–48; training stopped at practical utility.
|
|||
|
|
- **`atomicity` is measurement-limited.** LLM judges score a large FT win, but blind humans rated it a
|
|||
|
|
near-tie (27/30 ties) — the judge win is **not human-confirmed**. Treat atomicity cautiously.
|
|||
|
|
- **`discriminative` is family-sensitive.** Human- and judge-confirmed vs the 14B, but vs raw 7B it
|
|||
|
|
was **not significant** under a cross-family judge.
|
|||
|
|
- **The human eval is author-blind, not third-party.** I rated blind (arm-anonymized, order
|
|||
|
|
randomized), which removes judge-family bias, but I built the model and know its style — so it is
|
|||
|
|
**not a fully independent** human evaluation. (It was conservative: 88/150 ratings were ties.)
|
|||
|
|
- **The 5 principles are my own rubric**, not a standard benchmark; absolute faithfulness is low on
|
|||
|
|
hard prompts (the claim is "more faithful than the baselines," paired — not "faithful" in absolute
|
|||
|
|
terms).
|
|||
|
|
- **Downstream utility (retrieval/QA) is not evaluated.** Scope is distillation quality only.
|
|||
|
|
|
|||
|
|
## Usage
|
|||
|
|
|
|||
|
|
Deployment target is **Ollama Q4_K_M**, called via an OpenAI-compatible endpoint
|
|||
|
|
(`http://localhost:11434/v1`) — the same path codeatrium uses. Prompt the model with your distillation
|
|||
|
|
instruction + schema + the conversation, and read back the structured JSON (greedy / temperature 0 for
|
|||
|
|
determinism). The 5-principle behavior is strongest on **field-structured** schemas.
|
|||
|
|
|
|||
|
|
> Technical note: this repo ships the **fully merged weights** (base ⊕ SFT-LoRA ⊕ ORPO-LoRA already
|
|||
|
|
> baked in) — load it directly like any Qwen2.5-7B checkpoint, no adapter reconstruction needed.
|
|||
|
|
> Internally it was trained as `stock Qwen ⊕ SFT-LoRA → merge → ⊕ ORPO-LoRA → merge`; that provenance
|
|||
|
|
> only matters if you're retraining from an intermediate checkpoint, not for inference.
|
|||
|
|
|
|||
|
|
## References
|
|||
|
|
|
|||
|
|
- **ProMem** — proactive memory verification & completion. arXiv:2601.04463
|
|||
|
|
- **SPAR** — lexical-fidelity gap of dense retrievers. arXiv:2110.06918 (EMNLP-Findings 2022)
|
|||
|
|
- **Searchat** — structured distillation / surviving vocabulary. arXiv:2603.13017
|
|||
|
|
- **TriMem** — multi-granularity memory (segment / atomic fact / profile). arXiv:2605.19952
|
|||
|
|
- **HaluMem** — memory hallucination benchmark. arXiv:2511.03506
|
|||
|
|
- **GAAMA** — graph-augmented associative memory (megahub avoidance). arXiv:2603.27910 /
|
|||
|
|
github.com/swarna-kpaul/gaama
|
|||
|
|
- **HippoRAG** — neurobiologically-inspired long-term memory / entity-centric KG retrieval
|
|||
|
|
(hub-concentration evidence). arXiv:2405.14831
|