初始化项目，由ModelHub XC社区提供模型
Model: sennaLLMLearner/qwen2.5-7b-memory-distiller Source: Original Platform
This commit is contained in:
ModelHub XC
135
README.md
Normal file
135
README.md
Normal file
@@ -0,0 +1,135 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
base_model: Qwen/Qwen2.5-7B-Instruct
|
||||
datasets:
|
||||
- allenai/WildChat-1M
|
||||
language:
|
||||
- en
|
||||
- ja
|
||||
pipeline_tag: text-generation
|
||||
library_name: transformers
|
||||
tags:
|
||||
- memory
|
||||
- distillation
|
||||
- orpo
|
||||
- merged
|
||||
- qwen2.5
|
||||
- structured-output
|
||||
- json
|
||||
- edge
|
||||
- personal-project
|
||||
---
|
||||
|
||||
# Conversation Memory Distiller (Qwen2.5-7B)
|
||||
|
||||
A small model fine-tuned to turn a conversation log into clean, searchable **structured-memory
|
||||
JSON** for later recall, running **locally on an edge device**. Fine-tuned from
|
||||
`Qwen/Qwen2.5-7B-Instruct` via **SFT warmup → ORPO**, distilling a DeepSeek-V3 teacher, and
|
||||
optimized to follow **5 content principles** (below) when compressing a conversation into memory.
|
||||
|
||||
> **This is a structured-distillation (memory-summarization) model, not a general summarizer.**
|
||||
> It is meant to run **locally on an edge device** and emit schema-conformant memory objects.
|
||||
|
||||
## Why this exists
|
||||
|
||||
This model powers the local memory layer of **[codeatrium](https://github.com/senna-lang/Codeatrium)**, a personal-memory app: it distills
|
||||
conversations into searchable memories. The goal was a **local / free / private** 7B that runs on
|
||||
an edge device (via Ollama Q4_K_M) and could replace the larger distiller previously used in that
|
||||
role. It's a **personal project**: the dataset and evaluation are small-scale, the "5 principles"
|
||||
are **a custom rubric** drawn from the memory-distillation literature, and **training was stopped
|
||||
pragmatically once it reached practical quality** (2 ORPO epochs), not pushed to convergence.
|
||||
Metrics and acceptance gates were **pre-registered before running** the final evals, and only
|
||||
what survived is reported (see [How I know](#how-i-know)).
|
||||
|
||||
## Intended use / Out of scope
|
||||
|
||||
- **Intended**: distilling a conversation (user+assistant turns) into a **structured memory object**
|
||||
(JSON/record with fields) for later **search / recall**, running locally.
|
||||
- **Out of scope**: general-purpose summarization; downstream retrieval/QA quality (not evaluated
|
||||
here — only distillation quality is); absolute faithfulness guarantees.
|
||||
|
||||
## Learning signal: the 5 core principles
|
||||
|
||||
The model is trained with these 5 principles as the preference signal (chosen/rejected). Each is a
|
||||
content-level best practice backed by external work on memory systems and retrieval.
|
||||
|
||||
| Principle | Definition | Primary source | Basis (key figure) | How it's measured |
|
||||
|---|---|---|---|---|
|
||||
| **completeness** (recall) | capture the source's important specifics without dropping them | ProMem (arXiv:2601.04463) | verification+completion raises memory integrity 54→74%; +30.89pp vs Mem0 on HaluMem; **+10.74pt transfer on Llama3-8B** (the one principle confirmed to help at small scale) | deterministic: specifics recall against the source, no LLM judge |
|
||||
| **lexical_fidelity** | keep numbers, commands, identifiers **verbatim**, not paraphrased | SPAR (arXiv:2110.06918) / Searchat (arXiv:2603.13017) | dense retrievers miss rare entities / salient phrases that BM25 catches (**lexical-fidelity gap**); Searchat retains 96.8% of query vocabulary after distillation | deterministic: verbatim-retention rate among captured specifics (correlated with completeness, not fully independent) |
|
||||
| **atomicity** | one memory = one fact; no cramming multiple facts together | TriMem (arXiv:2605.19952) / Mem0 | fact-centric atomic decomposition; coarse-only memory blocks deep reasoning across scattered facts | LLM judge only; no deterministic anchor |
|
||||
| **discriminative** | carries specific info that can be retrieved / told apart (not generic labels) | GAAMA (arXiv:2603.27910) / HippoRAG (arXiv:2405.14831) | generic labels ("programming") make megahub nodes that absorb retrieval mass (**megahub problem**); discriminative tags narrow retrieval | LLM judge only; no deterministic anchor |
|
||||
| **faithfulness** (precision) | do not fabricate specifics absent from the source | ProMem (arXiv:2601.04463) / HaluMem (arXiv:2511.03506) | stronger completion increases fabrication risk (the flip side of completeness); HaluMem measures hallucination | deterministic: specifics precision against the source, no LLM judge |
|
||||
|
||||
> **Eval note.** `completeness` and `faithfulness` are measured **without an LLM judge** as set
|
||||
> recall / precision of *specifics* (numbers, identifiers, commands, paths, quoted strings) against
|
||||
> the source. `lexical_fidelity` is a supplementary verbatim-retention rate (not independent of
|
||||
> completeness). `atomicity` and `discriminative` require structural/semantic judgment and are
|
||||
> measured only via LLM judges — they carry no deterministic anchor (see limitations).
|
||||
|
||||
## Training
|
||||
|
||||
- **Base**: `Qwen/Qwen2.5-7B-Instruct`. **Teacher**: DeepSeek-V3 (distillation target).
|
||||
- **Data**: technical conversations deterministically sampled from **WildChat-1M** (en+ja,
|
||||
toxic-filtered) — a **small** preference set (~hundreds of conversations / thousands of exchanges).
|
||||
- **Recipe**: **SFT warmup → merge → fresh LoRA (r16) → ORPO (2 epochs)**.
|
||||
|
||||
## How I know
|
||||
|
||||
I evaluated **distillation quality** (the 5 principles), *not* downstream retrieval. Metrics and
|
||||
acceptance gates were **pre-registered before the final runs**.
|
||||
|
||||
**It beats the incumbent on the deployment task.** On my codeatrium production path (all arms
|
||||
**Ollama Q4_K_M**, 48 independent real conversations), this 7B outperforms both
|
||||
`qwen2.5-coder:14b` (my previous distiller) and raw `qwen2.5:7b`. The win is **not judge-dependent** —
|
||||
it holds under a same-family judge (Qwen2.5-72B), a **different-family judge** (Llama-3.3-70B), a
|
||||
**judge-free specifics metric**, and a **blind human rating** (n=30, FT vs the 14B):
|
||||
|
||||
- **completeness** is confirmed by *all four* methods (human net **+0.63**, p≈7e-5) — the flagship result.
|
||||
- **discriminative** (+0.47) and **lexical_fidelity** (+0.37) are also human-confirmed vs the 14B.
|
||||
|
||||
**It generalizes to unseen *structured* schemas — but not to everything.** On 5 held-out schemas it
|
||||
never trained on (measured judge-free), it beats raw 7B on **field-structured** formats (nested JSON,
|
||||
typed record, entity-relation graph — all significant) but only **ties** on flat / hard-length-capped
|
||||
formats (flat Markdown bullets, ≤12-word YAML). So the honest claim is: **generalizes to unseen
|
||||
field-structured schemas; no edge on flat / heavily-constrained formats.** (codeatrium's schema is
|
||||
field-structured, consistent with the deployment win.)
|
||||
|
||||
## Limitations (read this)
|
||||
|
||||
- **Small-scale & personal.** Small dataset; evals of n=30–48; training stopped at practical utility.
|
||||
- **`atomicity` is measurement-limited.** LLM judges score a large FT win, but blind humans rated it a
|
||||
near-tie (27/30 ties) — the judge win is **not human-confirmed**. Treat atomicity cautiously.
|
||||
- **`discriminative` is family-sensitive.** Human- and judge-confirmed vs the 14B, but vs raw 7B it
|
||||
was **not significant** under a cross-family judge.
|
||||
- **The human eval is author-blind, not third-party.** I rated blind (arm-anonymized, order
|
||||
randomized), which removes judge-family bias, but I built the model and know its style — so it is
|
||||
**not a fully independent** human evaluation. (It was conservative: 88/150 ratings were ties.)
|
||||
- **The 5 principles are my own rubric**, not a standard benchmark; absolute faithfulness is low on
|
||||
hard prompts (the claim is "more faithful than the baselines," paired — not "faithful" in absolute
|
||||
terms).
|
||||
- **Downstream utility (retrieval/QA) is not evaluated.** Scope is distillation quality only.
|
||||
|
||||
## Usage
|
||||
|
||||
Deployment target is **Ollama Q4_K_M**, called via an OpenAI-compatible endpoint
|
||||
(`http://localhost:11434/v1`) — the same path codeatrium uses. Prompt the model with your distillation
|
||||
instruction + schema + the conversation, and read back the structured JSON (greedy / temperature 0 for
|
||||
determinism). The 5-principle behavior is strongest on **field-structured** schemas.
|
||||
|
||||
> Technical note: this repo ships the **fully merged weights** (base ⊕ SFT-LoRA ⊕ ORPO-LoRA already
|
||||
> baked in) — load it directly like any Qwen2.5-7B checkpoint, no adapter reconstruction needed.
|
||||
> Internally it was trained as `stock Qwen ⊕ SFT-LoRA → merge → ⊕ ORPO-LoRA → merge`; that provenance
|
||||
> only matters if you're retraining from an intermediate checkpoint, not for inference.
|
||||
|
||||
## References
|
||||
|
||||
- **ProMem** — proactive memory verification & completion. arXiv:2601.04463
|
||||
- **SPAR** — lexical-fidelity gap of dense retrievers. arXiv:2110.06918 (EMNLP-Findings 2022)
|
||||
- **Searchat** — structured distillation / surviving vocabulary. arXiv:2603.13017
|
||||
- **TriMem** — multi-granularity memory (segment / atomic fact / profile). arXiv:2605.19952
|
||||
- **HaluMem** — memory hallucination benchmark. arXiv:2511.03506
|
||||
- **GAAMA** — graph-augmented associative memory (megahub avoidance). arXiv:2603.27910 /
|
||||
github.com/swarna-kpaul/gaama
|
||||
- **HippoRAG** — neurobiologically-inspired long-term memory / entity-centric KG retrieval
|
||||
(hub-concentration evidence). arXiv:2405.14831
|
||||
Reference in New Issue
Block a user