weft-lineage-extractor-1.5b/README.md

---
license: other
license_name: weft-research
license_link: https://github.com/wallfacers/data-weave
pipeline_tag: text-generation
library_name: transformers
base_model: Qwen/Qwen2.5-Coder-1.5B-Instruct
datasets:
- wallfacers/weft-script-lineage-synth
language:
- en
tags:
- research-artifact
- negative-result
- memorization
- domain-shift
- data-lineage
- etl
- lora
model-index:
- name: weft-lineage-extractor-1.5b
  results:
  - task:
      type: table-level-lineage-extraction
      name: ETL table-level data-lineage extraction
    dataset:
      type: synthetic-etl
      name: synthetic held-out (structural-form isolated)
    metrics:
    - type: precision
      value: 0.995
      name: Table precision (synthetic held-out)
  - task:
      type: table-level-lineage-extraction
      name: ETL table-level data-lineage extraction
    dataset:
      type: real-github-etl
      name: real GitHub ETL (human gold, n=139)
    metrics:
    - type: precision
      value: 0.270
      name: Table precision (real, out-of-distribution)
    - type: accuracy
      value: 0.496
      name: Read/write direction accuracy (real)
widget:
- example_title: Clean literal case (works)
  text: |
    task_type: PYTHON
    script:
    import psycopg2
    cur.execute("SELECT id, name FROM users WHERE active = 1")
    cur.execute("INSERT INTO user_summary (user_id) VALUES (%s)", rows)
---

# weft-lineage-extractor-1.5b

> ## ⚠️ RESEARCH ARTIFACT — a NEGATIVE RESULT about *synthetic-only* training. Not a production tool.
>
> ### ✅ Resolved: real-corpus training fixes this. If you want a **usable** lineage extractor, use **[weft-lineage-extractor-3b](https://huggingface.co/wallfacers/weft-lineage-extractor-3b)** — same task, trained on **real** scripts, real precision **0.33 → 0.64**, memorization leak gone.

A 1.5B model LoRA-fine-tuned **only on synthetic ETL scripts** to extract table-level data
lineage. On its **synthetic** held-out set it looks near-perfect (**precision 0.995**). On
**real GitHub ETL scripts it collapses** (precision **0.27**), and a large share of its
mistakes are **verbatim table names memorized from the synthetic training pool** (**22–40%**
of hallucinations, depending on language). It is published so the failure — a systematic
pathology of *synthetic-only training* — is reproducible and citable, and so the real-corpus
resolution ([3B](https://huggingface.co/wallfacers/weft-lineage-extractor-3b)) has a baseline.

**Takeaway:** synthetic-benchmark scores for structured-extraction models can be *severely*
optimistic. A model can ace a held-out synthetic split by *memorizing the generator's
vocabulary*, then emit those memorized names on real, out-of-distribution inputs.

- **Base:** [Qwen/Qwen2.5-Coder-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct)
- **Training data:** 10,000 **synthetic** ETL scripts (Python/Shell, 9 structural forms) — no real scripts in training.
- **Companion artifacts:** 0.5B / 3B scale points, a Scala/Java (JVM) variant, and the real-corpus 3B resolution.

---

## The headline: synthetic looks great, real does not

Same model, table-level metrics, identical extraction convention ("Convention A": label a table
only if its literal name appears in an executable read/write statement; ignore dynamic names,
file paths, temp views, comments, config-driven jobs).

| Evaluation set | precision | direction acc. | hallucination |
|---|---|---|---|
| **Synthetic held-out** (600, structural-form isolated) | **0.995** | **0.995** | 0.001 |
| **Real GitHub ETL** (139 scripts, human gold) | **0.270** | **0.496** | 0.153 |

**Four-way comparison on the real Python/Shell set** (n=139, non-empty gold 59):

| extractor | precision | hallucination | recall (non-∅) | direction (non-∅) |
|---|---|---|---|---|
| **this model (synthetic 1.5B)** | 0.270 | 0.153 | 0.618 | 0.496 |
| Qwen-Max (general LLM) | 0.327 | 0.301 | 0.939 | 0.872 |
| Claude (general LLM) | 0.542 | 0.134 | 0.806 | 0.730 |
| regex baseline | 0.166 | 0.000 | 0.473 | 0.397 |
| **real-corpus 3B (the resolution)** | **0.64** | low | 0.63 | — |

---

## Why it fails: memorization leak

A **hallucination** = a predicted table name that is neither in the gold nor literally present in
the script. We check how many are **verbatim** names from the synthetic training pool, or share
its **shape** (`schema.schema_base_suffix`, e.g. `dws.dws_member_point_di`).

| set | hallucinations | verbatim training-pool names | synthetic-shaped |
|---|---|---|---|
| Python/Shell real | 76 | **17 (22.4%)** | 19 (25.0%) |
| JVM (Scala/Java) real | 98 | **40 (40.8%)** | 49 (50.0%) |

Given a real script it cannot parse, the model **falls back to reciting training table names**.
This is the negative result, and it is **gold-independent**.

### Scale & cross-language

| scale | synthetic prec | real prec | real direction | **verbatim leak** |
|---|---|---|---|---|
| 0.5B | 0.994 | 0.243 | 0.369 | **37.4%** |
| 1.5B (this) | 0.995 | 0.270 | 0.496 | **22.4%** |
| 3B (synthetic) | 0.988 | 0.325 | 0.468 | **10.9%** |
| 1.5B + JVM, real JVM eval | ~0.99 | 0.165 | 0.418 | **40.8%** |
| **3B, real corpus** | — | **0.64** | — | **~0** |

Memorization leak shrinks monotonically with model size (a capacity problem), but direction
confusion does not improve with scale and the failure reproduces across languages. "More
synthetic data" does not close the gap — **real training data does** (bottom row).

---

## Intended use

- ✅ **Reproducing / studying** the synthetic-only-training memorization-leak failure.
- ✅ A **baseline** for abstention, real-data augmentation, or leak-mitigation research.
- ❌ **Not** for production lineage — use [weft-lineage-extractor-3b](https://huggingface.co/wallfacers/weft-lineage-extractor-3b) instead.

---

## Prompt format & quick start

System prompt (exact — must match training verbatim):

```
You are a data lineage extractor for ETL scripts. Given a PYTHON or SHELL task
script, output ONLY a JSON object {"reads": [...], "writes": [...]} where each
item is {"table": str, "columns": [str] or null}. Rules: include a table only if
its literal name appears in the script text; ignore dynamically-built table names,
commented-out SQL, and SQL that is merely printed or logged; if nothing is read or
written, output {"reads": [], "writes": []}.
```

```python
import json, re, torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = "wallfacers/weft-lineage-extractor-1.5b"
tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16, device_map="auto").eval()

SYSTEM = ("You are a data lineage extractor for ETL scripts. Given a PYTHON or SHELL task "
          "script, output ONLY a JSON object {\"reads\": [...], \"writes\": [...]} where each "
          "item is {\"table\": str, \"columns\": [str] or null}. Rules: include a table only if "
          "its literal name appears in the script text; ignore dynamically-built table names, "
          "commented-out SQL, and SQL that is merely printed or logged; if nothing is read or "
          "written, output {\"reads\": [], \"writes\": []}.")

def extract(task_type, script, max_new_tokens=256):
    msgs = [{"role": "system", "content": SYSTEM},
            {"role": "user", "content": f"task_type: {task_type}\nscript:\n{script}"}]
    inp = tok.apply_chat_template(msgs, add_generation_prompt=True, tokenize=True,
                                  return_dict=True, return_tensors="pt").to(model.device)
    with torch.no_grad():
        out = model.generate(**inp, max_new_tokens=max_new_tokens, do_sample=False,
                             pad_token_id=tok.pad_token_id or tok.eos_token_id)
    raw = tok.decode(out[0][inp["input_ids"].shape[1]:], skip_special_tokens=True).strip()
    m = re.search(r"\{.*\}", raw, re.DOTALL)
    return json.loads(m.group(0)) if m else {"reads": [], "writes": []}

print(extract("PYTHON", 'cur.execute("SELECT * FROM orders WHERE status = \'pending\'")'))
# -> {"reads": [{"table": "orders", "columns": null}], "writes": []}
```

---

## Training

| Parameter | Value |
|---|---|
| Base model | Qwen/Qwen2.5-Coder-1.5B-Instruct |
| Method | LoRA (r=16, α=32, dropout=0.05; q/k/v/o/gate/up/down_proj) |
| Epochs / LR | 2 / 2e-4 cosine, 3% warmup |
| Effective batch / max len | 16 (2×8 grad-accum) / 2048 |
| Precision / hardware | bfloat16 / single 12 GB GPU |
| Training data | 10,000 **synthetic** ETL scripts (9 structural forms) — **zero real scripts** |
| Seed | 20260703 (reproducible) |

---

## Limitations & honest disclosures

- **Not a production tool.** Real-world precision ~0.27; direction ~coin-flip. Use the 3B real-corpus model.
- **Literal-only by design:** dynamic names, commented/logged SQL, temp views, config-driven jobs are out of scope.
- **Evaluation gold** is human-adjudicated under Convention A; real sets are small (Python/Shell n=139; JVM n=141). The **leak metric is gold-independent** (verbatim 40.4%→40.8% on JVM under full re-adjudication).
- **Column-level** output exists in the schema but is best-effort; evaluated claims are table-level.

---

## Links & citation

- **Real-corpus resolution:** [weft-lineage-extractor-3b](https://huggingface.co/wallfacers/weft-lineage-extractor-3b)
- **Dataset (synthetic + eval/leak reports):** [wallfacers/weft-script-lineage-synth](https://huggingface.co/datasets/wallfacers/weft-script-lineage-synth)
- **Platform:** [Weft (data-weave)](https://github.com/wallfacers/data-weave)

```bibtex
@misc{weft-lineage-negresult-2026,
  author       = {{Weft Contributors}},
  title        = {{Synthetic-only training induces memorization leak in small
                   models for ETL data-lineage extraction: a negative result}},
  year         = 2026,
  publisher    = {{Hugging Face}},
  howpublished = {{\url{https://huggingface.co/wallfacers/weft-lineage-extractor-1.5b}}},
}
```

Trained with [TRL](https://huggingface.co/docs/trl) + [PEFT](https://huggingface.co/docs/peft).