94 lines
3.6 KiB
Markdown
94 lines
3.6 KiB
Markdown
|
---
|
||
|
|
license: other
|
||
|
|
license_name: weft-research
|
||
|
|
license_link: https://github.com/wallfacers/data-weave
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
library_name: transformers
|
||
|
|
base_model: Qwen/Qwen2.5-Coder-0.5B-Instruct
|
||
|
|
datasets:
|
||
|
|
- wallfacers/weft-script-lineage-synth
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
tags:
|
||
|
|
- research-artifact
|
||
|
|
- negative-result
|
||
|
|
- memorization
|
||
|
|
- domain-shift
|
||
|
|
- data-lineage
|
||
|
|
- etl
|
||
|
|
- lora
|
||
|
|
model-index:
|
||
|
|
- name: weft-lineage-extractor-0.5b
|
||
|
|
results:
|
||
|
|
- task:
|
||
|
|
type: table-level-lineage-extraction
|
||
|
|
name: ETL table-level data-lineage extraction
|
||
|
|
dataset:
|
||
|
|
type: synthetic-etl
|
||
|
|
name: synthetic held-out (structural-form isolated)
|
||
|
|
metrics:
|
||
|
|
- type: precision
|
||
|
|
value: 0.994
|
||
|
|
name: Table precision (synthetic held-out)
|
||
|
|
- task:
|
||
|
|
type: table-level-lineage-extraction
|
||
|
|
name: ETL table-level data-lineage extraction
|
||
|
|
dataset:
|
||
|
|
type: real-github-etl
|
||
|
|
name: real GitHub ETL (human gold)
|
||
|
|
metrics:
|
||
|
|
- type: precision
|
||
|
|
value: 0.243
|
||
|
|
name: Table precision (real, out-of-distribution)
|
||
|
|
---
|
||
|
|
|
||
|
|
# weft-lineage-extractor-0.5b — smallest scale point of a NEGATIVE RESULT
|
||
|
|
|
||
|
|
> ## ⚠️ RESEARCH ARTIFACT — the 0.5B point of a *synthetic-only* training study. Not a production tool.
|
||
|
|
>
|
||
|
|
> ### ✅ Resolved by real data: use **[weft-lineage-extractor-3b](https://huggingface.co/wallfacers/weft-lineage-extractor-3b)** (real corpus, real precision **0.64**). Full study: **[weft-lineage-extractor-1.5b](https://huggingface.co/wallfacers/weft-lineage-extractor-1.5b)**.
|
||
|
|
|
||
|
|
The **0.5B** point of a study showing that **synthetic-only training** induces a **verbatim
|
||
|
|
memorization leak** in small models for ETL table-lineage extraction. It is the smallest scale
|
||
|
|
point and shows the **worst** leak: near-perfect synthetic precision (0.994) collapses to
|
||
|
|
**0.243** on real GitHub scripts, with **37.4%** of hallucinations being table names recited
|
||
|
|
verbatim from the synthetic training pool.
|
||
|
|
|
||
|
|
Same recipe as the [1.5B main model](https://huggingface.co/wallfacers/weft-lineage-extractor-1.5b)
|
||
|
|
(LoRA on Qwen2.5-Coder-Instruct, Python/Shell synthetic ETL scripts, zero real scripts); only the
|
||
|
|
base size differs.
|
||
|
|
|
||
|
|
## This point's numbers (table-level, Convention A)
|
||
|
|
|
||
|
|
| metric | synthetic held-out | real GitHub ETL |
|
||
|
|
|---|---|---|
|
||
|
|
| precision | 0.994 | **0.243** |
|
||
|
|
| direction accuracy | — | **0.369** |
|
||
|
|
| **verbatim memorization leak** | — | **37.4%** |
|
||
|
|
|
||
|
|
## Where it fits (scale curve)
|
||
|
|
|
||
|
|
| scale | real precision | real direction | verbatim leak |
|
||
|
|
|---|---|---|---|
|
||
|
|
| **0.5B (this)** | **0.243** | **0.369** | **37.4%** |
|
||
|
|
| 1.5B (main) | 0.270 | 0.496 | 22.4% |
|
||
|
|
| 3B (synthetic) | 0.325 | 0.468 | 10.9% |
|
||
|
|
| **3B (real corpus)** | **0.64** | — | **~0** |
|
||
|
|
|
||
|
|
Leak shrinks with scale (capacity), but only **real training data** closes the real-world gap
|
||
|
|
(bottom row). Direction confusion does not improve with size.
|
||
|
|
|
||
|
|
## Intended use
|
||
|
|
|
||
|
|
- ✅ Reproducing / studying the synthetic-training memorization-leak failure at minimal scale.
|
||
|
|
- ❌ Not for production lineage — use the [real-corpus 3B](https://huggingface.co/wallfacers/weft-lineage-extractor-3b).
|
||
|
|
|
||
|
|
## Usage, prompt format, training details, citation
|
||
|
|
|
||
|
|
Identical to the main model (this variant uses `task_type: PYTHON | SHELL`; everything else the same):
|
||
|
|
**[weft-lineage-extractor-1.5b](https://huggingface.co/wallfacers/weft-lineage-extractor-1.5b)**.
|
||
|
|
|
||
|
|
- **Dataset & reports:** [wallfacers/weft-script-lineage-synth](https://huggingface.co/datasets/wallfacers/weft-script-lineage-synth)
|
||
|
|
- **Base model:** [Qwen/Qwen2.5-Coder-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-0.5B-Instruct)
|
||
|
|
- **Platform:** [Weft (data-weave)](https://github.com/wallfacers/data-weave)
|