weft-lineage-extractor-1.5b/README.md

license, license_name, license_link, pipeline_tag, library_name, base_model, datasets, language, tags, model-index, widget

license	license_name	license_link	pipeline_tag	library_name	base_model	datasets	language	tags	model-index	widget
other	weft-research	https://github.com/wallfacers/data-weave	text-generation	transformers	Qwen/Qwen2.5-Coder-1.5B-Instruct	wallfacers/weft-script-lineage-synth	en	research-artifact negative-result memorization domain-shift data-lineage etl lora	name results weft-lineage-extractor-1.5b task dataset metrics type name table-level-lineage-extraction ETL table-level data-lineage extraction type name synthetic-etl synthetic held-out (structural-form isolated) type value name precision 0.995 Table precision (synthetic held-out) task dataset metrics type name table-level-lineage-extraction ETL table-level data-lineage extraction type name real-github-etl real GitHub ETL (human gold, n=139) type value name precision 0.270 Table precision (real, out-of-distribution) type value name accuracy 0.496 Read/write direction accuracy (real)	example_title text Clean literal case (works) task_type: PYTHON script: import psycopg2 cur.execute("SELECT id, name FROM users WHERE active = 1") cur.execute("INSERT INTO user_summary (user_id) VALUES (%s)", rows)

weft-lineage-extractor-1.5b

⚠️ RESEARCH ARTIFACT — a NEGATIVE RESULT about synthetic-only training. Not a production tool.

✅ Resolved: real-corpus training fixes this. If you want a usable lineage extractor, use weft-lineage-extractor-3b — same task, trained on real scripts, real precision 0.33 → 0.64, memorization leak gone.

A 1.5B model LoRA-fine-tuned only on synthetic ETL scripts to extract table-level data lineage. On its synthetic held-out set it looks near-perfect (precision 0.995). On real GitHub ETL scripts it collapses (precision 0.27), and a large share of its mistakes are verbatim table names memorized from the synthetic training pool (22–40% of hallucinations, depending on language). It is published so the failure — a systematic pathology of synthetic-only training — is reproducible and citable, and so the real-corpus resolution (3B) has a baseline.

Takeaway: synthetic-benchmark scores for structured-extraction models can be severely optimistic. A model can ace a held-out synthetic split by memorizing the generator's vocabulary, then emit those memorized names on real, out-of-distribution inputs.

• Base: Qwen/Qwen2.5-Coder-1.5B-Instruct
• Training data: 10,000 synthetic ETL scripts (Python/Shell, 9 structural forms) — no real scripts in training.
• Companion artifacts: 0.5B / 3B scale points, a Scala/Java (JVM) variant, and the real-corpus 3B resolution.

---

The headline: synthetic looks great, real does not

Same model, table-level metrics, identical extraction convention ("Convention A": label a table only if its literal name appears in an executable read/write statement; ignore dynamic names, file paths, temp views, comments, config-driven jobs).

Evaluation set	precision	direction acc.	hallucination
Synthetic held-out (600, structural-form isolated)	0.995	0.995	0.001
Real GitHub ETL (139 scripts, human gold)	0.270	0.496	0.153

Four-way comparison on the real Python/Shell set (n=139, non-empty gold 59):

extractor	precision	hallucination	recall (non-∅)	direction (non-∅)
this model (synthetic 1.5B)	0.270	0.153	0.618	0.496
Qwen-Max (general LLM)	0.327	0.301	0.939	0.872
Claude (general LLM)	0.542	0.134	0.806	0.730
regex baseline	0.166	0.000	0.473	0.397
real-corpus 3B (the resolution)	0.64	low	0.63	—

---

Why it fails: memorization leak

A hallucination = a predicted table name that is neither in the gold nor literally present in the script. We check how many are verbatim names from the synthetic training pool, or share its shape (schema.schema_base_suffix, e.g. dws.dws_member_point_di).

set	hallucinations	verbatim training-pool names	synthetic-shaped
Python/Shell real	76	17 (22.4%)	19 (25.0%)
JVM (Scala/Java) real	98	40 (40.8%)	49 (50.0%)

Given a real script it cannot parse, the model falls back to reciting training table names. This is the negative result, and it is gold-independent.

Scale & cross-language

scale	synthetic prec	real prec	real direction	verbatim leak
0.5B	0.994	0.243	0.369	37.4%
1.5B (this)	0.995	0.270	0.496	22.4%
3B (synthetic)	0.988	0.325	0.468	10.9%
1.5B + JVM, real JVM eval	~0.99	0.165	0.418	40.8%
3B, real corpus	—	0.64	—	~0

Memorization leak shrinks monotonically with model size (a capacity problem), but direction confusion does not improve with scale and the failure reproduces across languages. "More synthetic data" does not close the gap — real training data does (bottom row).

---

Intended use

• ✅ Reproducing / studying the synthetic-only-training memorization-leak failure.
• ✅ A baseline for abstention, real-data augmentation, or leak-mitigation research.
• ❌ Not for production lineage — use weft-lineage-extractor-3b instead.

---

Prompt format & quick start

System prompt (exact — must match training verbatim):

You are a data lineage extractor for ETL scripts. Given a PYTHON or SHELL task
script, output ONLY a JSON object {"reads": [...], "writes": [...]} where each
item is {"table": str, "columns": [str] or null}. Rules: include a table only if
its literal name appears in the script text; ignore dynamically-built table names,
commented-out SQL, and SQL that is merely printed or logged; if nothing is read or
written, output {"reads": [], "writes": []}.

import json, re, torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = "wallfacers/weft-lineage-extractor-1.5b"
tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16, device_map="auto").eval()

SYSTEM = ("You are a data lineage extractor for ETL scripts. Given a PYTHON or SHELL task "
          "script, output ONLY a JSON object {\"reads\": [...], \"writes\": [...]} where each "
          "item is {\"table\": str, \"columns\": [str] or null}. Rules: include a table only if "
          "its literal name appears in the script text; ignore dynamically-built table names, "
          "commented-out SQL, and SQL that is merely printed or logged; if nothing is read or "
          "written, output {\"reads\": [], \"writes\": []}.")

def extract(task_type, script, max_new_tokens=256):
    msgs = [{"role": "system", "content": SYSTEM},
            {"role": "user", "content": f"task_type: {task_type}\nscript:\n{script}"}]
    inp = tok.apply_chat_template(msgs, add_generation_prompt=True, tokenize=True,
                                  return_dict=True, return_tensors="pt").to(model.device)
    with torch.no_grad():
        out = model.generate(**inp, max_new_tokens=max_new_tokens, do_sample=False,
                             pad_token_id=tok.pad_token_id or tok.eos_token_id)
    raw = tok.decode(out[0][inp["input_ids"].shape[1]:], skip_special_tokens=True).strip()
    m = re.search(r"\{.*\}", raw, re.DOTALL)
    return json.loads(m.group(0)) if m else {"reads": [], "writes": []}

print(extract("PYTHON", 'cur.execute("SELECT * FROM orders WHERE status = \'pending\'")'))
# -> {"reads": [{"table": "orders", "columns": null}], "writes": []}

---

Training

Parameter	Value
Base model	Qwen/Qwen2.5-Coder-1.5B-Instruct
Method	LoRA (r=16, α=32, dropout=0.05; q/k/v/o/gate/up/down_proj)
Epochs / LR	2 / 2e-4 cosine, 3% warmup
Effective batch / max len	16 (2×8 grad-accum) / 2048
Precision / hardware	bfloat16 / single 12 GB GPU
Training data	10,000 synthetic ETL scripts (9 structural forms) — zero real scripts
Seed	20260703 (reproducible)

---

Limitations & honest disclosures

• Not a production tool. Real-world precision ~0.27; direction ~coin-flip. Use the 3B real-corpus model.
• Literal-only by design: dynamic names, commented/logged SQL, temp views, config-driven jobs are out of scope.
• Evaluation gold is human-adjudicated under Convention A; real sets are small (Python/Shell n=139; JVM n=141). The leak metric is gold-independent (verbatim 40.4%→40.8% on JVM under full re-adjudication).
• Column-level output exists in the schema but is best-effort; evaluated claims are table-level.

---

Links & citation

• Real-corpus resolution: weft-lineage-extractor-3b
• Dataset (synthetic + eval/leak reports): wallfacers/weft-script-lineage-synth
• Platform: Weft (data-weave)

@misc{weft-lineage-negresult-2026,
  author       = {{Weft Contributors}},
  title        = {{Synthetic-only training induces memorization leak in small
                   models for ETL data-lineage extraction: a negative result}},
  year         = 2026,
  publisher    = {{Hugging Face}},
  howpublished = {{\url{https://huggingface.co/wallfacers/weft-lineage-extractor-1.5b}}},
}

Trained with TRL + PEFT.