228 lines
9.8 KiB
Markdown
228 lines
9.8 KiB
Markdown
---
|
||
license: other
|
||
license_name: weft-research
|
||
license_link: https://github.com/wallfacers/data-weave
|
||
pipeline_tag: text-generation
|
||
library_name: transformers
|
||
base_model: Qwen/Qwen2.5-Coder-1.5B-Instruct
|
||
datasets:
|
||
- wallfacers/weft-script-lineage-synth
|
||
language:
|
||
- en
|
||
tags:
|
||
- research-artifact
|
||
- negative-result
|
||
- memorization
|
||
- domain-shift
|
||
- data-lineage
|
||
- etl
|
||
- lora
|
||
model-index:
|
||
- name: weft-lineage-extractor-1.5b
|
||
results:
|
||
- task:
|
||
type: table-level-lineage-extraction
|
||
name: ETL table-level data-lineage extraction
|
||
dataset:
|
||
type: synthetic-etl
|
||
name: synthetic held-out (structural-form isolated)
|
||
metrics:
|
||
- type: precision
|
||
value: 0.995
|
||
name: Table precision (synthetic held-out)
|
||
- task:
|
||
type: table-level-lineage-extraction
|
||
name: ETL table-level data-lineage extraction
|
||
dataset:
|
||
type: real-github-etl
|
||
name: real GitHub ETL (human gold, n=139)
|
||
metrics:
|
||
- type: precision
|
||
value: 0.270
|
||
name: Table precision (real, out-of-distribution)
|
||
- type: accuracy
|
||
value: 0.496
|
||
name: Read/write direction accuracy (real)
|
||
widget:
|
||
- example_title: Clean literal case (works)
|
||
text: |
|
||
task_type: PYTHON
|
||
script:
|
||
import psycopg2
|
||
cur.execute("SELECT id, name FROM users WHERE active = 1")
|
||
cur.execute("INSERT INTO user_summary (user_id) VALUES (%s)", rows)
|
||
---
|
||
|
||
# weft-lineage-extractor-1.5b
|
||
|
||
> ## ⚠️ RESEARCH ARTIFACT — a NEGATIVE RESULT about *synthetic-only* training. Not a production tool.
|
||
>
|
||
> ### ✅ Resolved: real-corpus training fixes this. If you want a **usable** lineage extractor, use **[weft-lineage-extractor-3b](https://huggingface.co/wallfacers/weft-lineage-extractor-3b)** — same task, trained on **real** scripts, real precision **0.33 → 0.64**, memorization leak gone.
|
||
|
||
A 1.5B model LoRA-fine-tuned **only on synthetic ETL scripts** to extract table-level data
|
||
lineage. On its **synthetic** held-out set it looks near-perfect (**precision 0.995**). On
|
||
**real GitHub ETL scripts it collapses** (precision **0.27**), and a large share of its
|
||
mistakes are **verbatim table names memorized from the synthetic training pool** (**22–40%**
|
||
of hallucinations, depending on language). It is published so the failure — a systematic
|
||
pathology of *synthetic-only training* — is reproducible and citable, and so the real-corpus
|
||
resolution ([3B](https://huggingface.co/wallfacers/weft-lineage-extractor-3b)) has a baseline.
|
||
|
||
**Takeaway:** synthetic-benchmark scores for structured-extraction models can be *severely*
|
||
optimistic. A model can ace a held-out synthetic split by *memorizing the generator's
|
||
vocabulary*, then emit those memorized names on real, out-of-distribution inputs.
|
||
|
||
- **Base:** [Qwen/Qwen2.5-Coder-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct)
|
||
- **Training data:** 10,000 **synthetic** ETL scripts (Python/Shell, 9 structural forms) — no real scripts in training.
|
||
- **Companion artifacts:** 0.5B / 3B scale points, a Scala/Java (JVM) variant, and the real-corpus 3B resolution.
|
||
|
||
---
|
||
|
||
## The headline: synthetic looks great, real does not
|
||
|
||
Same model, table-level metrics, identical extraction convention ("Convention A": label a table
|
||
only if its literal name appears in an executable read/write statement; ignore dynamic names,
|
||
file paths, temp views, comments, config-driven jobs).
|
||
|
||
| Evaluation set | precision | direction acc. | hallucination |
|
||
|---|---|---|---|
|
||
| **Synthetic held-out** (600, structural-form isolated) | **0.995** | **0.995** | 0.001 |
|
||
| **Real GitHub ETL** (139 scripts, human gold) | **0.270** | **0.496** | 0.153 |
|
||
|
||
**Four-way comparison on the real Python/Shell set** (n=139, non-empty gold 59):
|
||
|
||
| extractor | precision | hallucination | recall (non-∅) | direction (non-∅) |
|
||
|---|---|---|---|---|
|
||
| **this model (synthetic 1.5B)** | 0.270 | 0.153 | 0.618 | 0.496 |
|
||
| Qwen-Max (general LLM) | 0.327 | 0.301 | 0.939 | 0.872 |
|
||
| Claude (general LLM) | 0.542 | 0.134 | 0.806 | 0.730 |
|
||
| regex baseline | 0.166 | 0.000 | 0.473 | 0.397 |
|
||
| **real-corpus 3B (the resolution)** | **0.64** | low | 0.63 | — |
|
||
|
||
---
|
||
|
||
## Why it fails: memorization leak
|
||
|
||
A **hallucination** = a predicted table name that is neither in the gold nor literally present in
|
||
the script. We check how many are **verbatim** names from the synthetic training pool, or share
|
||
its **shape** (`schema.schema_base_suffix`, e.g. `dws.dws_member_point_di`).
|
||
|
||
| set | hallucinations | verbatim training-pool names | synthetic-shaped |
|
||
|---|---|---|---|
|
||
| Python/Shell real | 76 | **17 (22.4%)** | 19 (25.0%) |
|
||
| JVM (Scala/Java) real | 98 | **40 (40.8%)** | 49 (50.0%) |
|
||
|
||
Given a real script it cannot parse, the model **falls back to reciting training table names**.
|
||
This is the negative result, and it is **gold-independent**.
|
||
|
||
### Scale & cross-language
|
||
|
||
| scale | synthetic prec | real prec | real direction | **verbatim leak** |
|
||
|---|---|---|---|---|
|
||
| 0.5B | 0.994 | 0.243 | 0.369 | **37.4%** |
|
||
| 1.5B (this) | 0.995 | 0.270 | 0.496 | **22.4%** |
|
||
| 3B (synthetic) | 0.988 | 0.325 | 0.468 | **10.9%** |
|
||
| 1.5B + JVM, real JVM eval | ~0.99 | 0.165 | 0.418 | **40.8%** |
|
||
| **3B, real corpus** | — | **0.64** | — | **~0** |
|
||
|
||
Memorization leak shrinks monotonically with model size (a capacity problem), but direction
|
||
confusion does not improve with scale and the failure reproduces across languages. "More
|
||
synthetic data" does not close the gap — **real training data does** (bottom row).
|
||
|
||
---
|
||
|
||
## Intended use
|
||
|
||
- ✅ **Reproducing / studying** the synthetic-only-training memorization-leak failure.
|
||
- ✅ A **baseline** for abstention, real-data augmentation, or leak-mitigation research.
|
||
- ❌ **Not** for production lineage — use [weft-lineage-extractor-3b](https://huggingface.co/wallfacers/weft-lineage-extractor-3b) instead.
|
||
|
||
---
|
||
|
||
## Prompt format & quick start
|
||
|
||
System prompt (exact — must match training verbatim):
|
||
|
||
```
|
||
You are a data lineage extractor for ETL scripts. Given a PYTHON or SHELL task
|
||
script, output ONLY a JSON object {"reads": [...], "writes": [...]} where each
|
||
item is {"table": str, "columns": [str] or null}. Rules: include a table only if
|
||
its literal name appears in the script text; ignore dynamically-built table names,
|
||
commented-out SQL, and SQL that is merely printed or logged; if nothing is read or
|
||
written, output {"reads": [], "writes": []}.
|
||
```
|
||
|
||
```python
|
||
import json, re, torch
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
||
MODEL = "wallfacers/weft-lineage-extractor-1.5b"
|
||
tok = AutoTokenizer.from_pretrained(MODEL)
|
||
model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16, device_map="auto").eval()
|
||
|
||
SYSTEM = ("You are a data lineage extractor for ETL scripts. Given a PYTHON or SHELL task "
|
||
"script, output ONLY a JSON object {\"reads\": [...], \"writes\": [...]} where each "
|
||
"item is {\"table\": str, \"columns\": [str] or null}. Rules: include a table only if "
|
||
"its literal name appears in the script text; ignore dynamically-built table names, "
|
||
"commented-out SQL, and SQL that is merely printed or logged; if nothing is read or "
|
||
"written, output {\"reads\": [], \"writes\": []}.")
|
||
|
||
def extract(task_type, script, max_new_tokens=256):
|
||
msgs = [{"role": "system", "content": SYSTEM},
|
||
{"role": "user", "content": f"task_type: {task_type}\nscript:\n{script}"}]
|
||
inp = tok.apply_chat_template(msgs, add_generation_prompt=True, tokenize=True,
|
||
return_dict=True, return_tensors="pt").to(model.device)
|
||
with torch.no_grad():
|
||
out = model.generate(**inp, max_new_tokens=max_new_tokens, do_sample=False,
|
||
pad_token_id=tok.pad_token_id or tok.eos_token_id)
|
||
raw = tok.decode(out[0][inp["input_ids"].shape[1]:], skip_special_tokens=True).strip()
|
||
m = re.search(r"\{.*\}", raw, re.DOTALL)
|
||
return json.loads(m.group(0)) if m else {"reads": [], "writes": []}
|
||
|
||
print(extract("PYTHON", 'cur.execute("SELECT * FROM orders WHERE status = \'pending\'")'))
|
||
# -> {"reads": [{"table": "orders", "columns": null}], "writes": []}
|
||
```
|
||
|
||
---
|
||
|
||
## Training
|
||
|
||
| Parameter | Value |
|
||
|---|---|
|
||
| Base model | Qwen/Qwen2.5-Coder-1.5B-Instruct |
|
||
| Method | LoRA (r=16, α=32, dropout=0.05; q/k/v/o/gate/up/down_proj) |
|
||
| Epochs / LR | 2 / 2e-4 cosine, 3% warmup |
|
||
| Effective batch / max len | 16 (2×8 grad-accum) / 2048 |
|
||
| Precision / hardware | bfloat16 / single 12 GB GPU |
|
||
| Training data | 10,000 **synthetic** ETL scripts (9 structural forms) — **zero real scripts** |
|
||
| Seed | 20260703 (reproducible) |
|
||
|
||
---
|
||
|
||
## Limitations & honest disclosures
|
||
|
||
- **Not a production tool.** Real-world precision ~0.27; direction ~coin-flip. Use the 3B real-corpus model.
|
||
- **Literal-only by design:** dynamic names, commented/logged SQL, temp views, config-driven jobs are out of scope.
|
||
- **Evaluation gold** is human-adjudicated under Convention A; real sets are small (Python/Shell n=139; JVM n=141). The **leak metric is gold-independent** (verbatim 40.4%→40.8% on JVM under full re-adjudication).
|
||
- **Column-level** output exists in the schema but is best-effort; evaluated claims are table-level.
|
||
|
||
---
|
||
|
||
## Links & citation
|
||
|
||
- **Real-corpus resolution:** [weft-lineage-extractor-3b](https://huggingface.co/wallfacers/weft-lineage-extractor-3b)
|
||
- **Dataset (synthetic + eval/leak reports):** [wallfacers/weft-script-lineage-synth](https://huggingface.co/datasets/wallfacers/weft-script-lineage-synth)
|
||
- **Platform:** [Weft (data-weave)](https://github.com/wallfacers/data-weave)
|
||
|
||
```bibtex
|
||
@misc{weft-lineage-negresult-2026,
|
||
author = {{Weft Contributors}},
|
||
title = {{Synthetic-only training induces memorization leak in small
|
||
models for ETL data-lineage extraction: a negative result}},
|
||
year = 2026,
|
||
publisher = {{Hugging Face}},
|
||
howpublished = {{\url{https://huggingface.co/wallfacers/weft-lineage-extractor-1.5b}}},
|
||
}
|
||
```
|
||
|
||
Trained with [TRL](https://huggingface.co/docs/trl) + [PEFT](https://huggingface.co/docs/peft).
|