---
license: apache-2.0
base_model: Qwen/Qwen2.5-7B-Instruct
datasets:
  - allenai/WildChat-1M
language:
  - en
  - ja
pipeline_tag: text-generation
library_name: transformers
tags:
  - memory
  - distillation
  - orpo
  - merged
  - qwen2.5
  - structured-output
  - json
  - edge
  - personal-project
---

# Conversation Memory Distiller (Qwen2.5-7B)

A small model fine-tuned to turn a conversation log into clean, searchable **structured-memory
JSON** for later recall, running **locally on an edge device**. Fine-tuned from
`Qwen/Qwen2.5-7B-Instruct` via **SFT warmup → ORPO**, distilling a DeepSeek-V3 teacher, and
optimized to follow **5 content principles** (below) when compressing a conversation into memory.

> **This is a structured-distillation (memory-summarization) model, not a general summarizer.**
> It is meant to run **locally on an edge device** and emit schema-conformant memory objects.

## Why this exists

This model powers the local memory layer of **[codeatrium](https://github.com/senna-lang/Codeatrium)**, a personal-memory app: it distills
conversations into searchable memories. The goal was a **local / free / private** 7B that runs on
an edge device (via Ollama Q4_K_M) and could replace the larger distiller previously used in that
role. It's a **personal project**: the dataset and evaluation are small-scale, the "5 principles"
are **a custom rubric** drawn from the memory-distillation literature, and **training was stopped
pragmatically once it reached practical quality** (2 ORPO epochs), not pushed to convergence.
Metrics and acceptance gates were **pre-registered before running** the final evals, and only
what survived is reported (see [How I know](#how-i-know)).

## Intended use / Out of scope

- **Intended**: distilling a conversation (user+assistant turns) into a **structured memory object**
  (JSON/record with fields) for later **search / recall**, running locally.
- **Out of scope**: general-purpose summarization; downstream retrieval/QA quality (not evaluated
  here — only distillation quality is); absolute faithfulness guarantees.

## Learning signal: the 5 core principles

The model is trained with these 5 principles as the preference signal (chosen/rejected). Each is a
content-level best practice backed by external work on memory systems and retrieval.

| Principle | Definition | Primary source | Basis (key figure) | How it's measured |
|---|---|---|---|---|
| **completeness** (recall) | capture the source's important specifics without dropping them | ProMem (arXiv:2601.04463) | verification+completion raises memory integrity 54→74%; +30.89pp vs Mem0 on HaluMem; **+10.74pt transfer on Llama3-8B** (the one principle confirmed to help at small scale) | deterministic: specifics recall against the source, no LLM judge |
| **lexical_fidelity** | keep numbers, commands, identifiers **verbatim**, not paraphrased | SPAR (arXiv:2110.06918) / Searchat (arXiv:2603.13017) | dense retrievers miss rare entities / salient phrases that BM25 catches (**lexical-fidelity gap**); Searchat retains 96.8% of query vocabulary after distillation | deterministic: verbatim-retention rate among captured specifics (correlated with completeness, not fully independent) |
| **atomicity** | one memory = one fact; no cramming multiple facts together | TriMem (arXiv:2605.19952) / Mem0 | fact-centric atomic decomposition; coarse-only memory blocks deep reasoning across scattered facts | LLM judge only; no deterministic anchor |
| **discriminative** | carries specific info that can be retrieved / told apart (not generic labels) | GAAMA (arXiv:2603.27910) / HippoRAG (arXiv:2405.14831) | generic labels ("programming") make megahub nodes that absorb retrieval mass (**megahub problem**); discriminative tags narrow retrieval | LLM judge only; no deterministic anchor |
| **faithfulness** (precision) | do not fabricate specifics absent from the source | ProMem (arXiv:2601.04463) / HaluMem (arXiv:2511.03506) | stronger completion increases fabrication risk (the flip side of completeness); HaluMem measures hallucination | deterministic: specifics precision against the source, no LLM judge |

> **Eval note.** `completeness` and `faithfulness` are measured **without an LLM judge** as set
> recall / precision of *specifics* (numbers, identifiers, commands, paths, quoted strings) against
> the source. `lexical_fidelity` is a supplementary verbatim-retention rate (not independent of
> completeness). `atomicity` and `discriminative` require structural/semantic judgment and are
> measured only via LLM judges — they carry no deterministic anchor (see limitations).

## Training

- **Base**: `Qwen/Qwen2.5-7B-Instruct`. **Teacher**: DeepSeek-V3 (distillation target).
- **Data**: technical conversations deterministically sampled from **WildChat-1M** (en+ja,
  toxic-filtered) — a **small** preference set (~hundreds of conversations / thousands of exchanges).
- **Recipe**: **SFT warmup → merge → fresh LoRA (r16) → ORPO (2 epochs)**.

## How I know

I evaluated **distillation quality** (the 5 principles), *not* downstream retrieval. Metrics and
acceptance gates were **pre-registered before the final runs**.

**It beats the incumbent on the deployment task.** On my codeatrium production path (all arms
**Ollama Q4_K_M**, 48 independent real conversations), this 7B outperforms both
`qwen2.5-coder:14b` (my previous distiller) and raw `qwen2.5:7b`. The win is **not judge-dependent** —
it holds under a same-family judge (Qwen2.5-72B), a **different-family judge** (Llama-3.3-70B), a
**judge-free specifics metric**, and a **blind human rating** (n=30, FT vs the 14B):

- **completeness** is confirmed by *all four* methods (human net **+0.63**, p≈7e-5) — the flagship result.
- **discriminative** (+0.47) and **lexical_fidelity** (+0.37) are also human-confirmed vs the 14B.

**It generalizes to unseen *structured* schemas — but not to everything.** On 5 held-out schemas it
never trained on (measured judge-free), it beats raw 7B on **field-structured** formats (nested JSON,
typed record, entity-relation graph — all significant) but only **ties** on flat / hard-length-capped
formats (flat Markdown bullets, ≤12-word YAML). So the honest claim is: **generalizes to unseen
field-structured schemas; no edge on flat / heavily-constrained formats.** (codeatrium's schema is
field-structured, consistent with the deployment win.)

## Limitations (read this)

- **Small-scale & personal.** Small dataset; evals of n=30–48; training stopped at practical utility.
- **`atomicity` is measurement-limited.** LLM judges score a large FT win, but blind humans rated it a
  near-tie (27/30 ties) — the judge win is **not human-confirmed**. Treat atomicity cautiously.
- **`discriminative` is family-sensitive.** Human- and judge-confirmed vs the 14B, but vs raw 7B it
  was **not significant** under a cross-family judge.
- **The human eval is author-blind, not third-party.** I rated blind (arm-anonymized, order
  randomized), which removes judge-family bias, but I built the model and know its style — so it is
  **not a fully independent** human evaluation. (It was conservative: 88/150 ratings were ties.)
- **The 5 principles are my own rubric**, not a standard benchmark; absolute faithfulness is low on
  hard prompts (the claim is "more faithful than the baselines," paired — not "faithful" in absolute
  terms).
- **Downstream utility (retrieval/QA) is not evaluated.** Scope is distillation quality only.

## Usage

Deployment target is **Ollama Q4_K_M**, called via an OpenAI-compatible endpoint
(`http://localhost:11434/v1`) — the same path codeatrium uses. Prompt the model with your distillation
instruction + schema + the conversation, and read back the structured JSON (greedy / temperature 0 for
determinism). The 5-principle behavior is strongest on **field-structured** schemas.

> Technical note: this repo ships the **fully merged weights** (base ⊕ SFT-LoRA ⊕ ORPO-LoRA already
> baked in) — load it directly like any Qwen2.5-7B checkpoint, no adapter reconstruction needed.
> Internally it was trained as `stock Qwen ⊕ SFT-LoRA → merge → ⊕ ORPO-LoRA → merge`; that provenance
> only matters if you're retraining from an intermediate checkpoint, not for inference.

## References

- **ProMem** — proactive memory verification & completion. arXiv:2601.04463
- **SPAR** — lexical-fidelity gap of dense retrievers. arXiv:2110.06918 (EMNLP-Findings 2022)
- **Searchat** — structured distillation / surviving vocabulary. arXiv:2603.13017
- **TriMem** — multi-granularity memory (segment / atomic fact / profile). arXiv:2605.19952
- **HaluMem** — memory hallucination benchmark. arXiv:2511.03506
- **GAAMA** — graph-augmented associative memory (megahub avoidance). arXiv:2603.27910 /
  github.com/swarna-kpaul/gaama
- **HippoRAG** — neurobiologically-inspired long-term memory / entity-centric KG retrieval
  (hub-concentration evidence). arXiv:2405.14831