初始化项目，由ModelHub XC社区提供模型
Model: wallfacers/weft-lineage-extractor-1.5b Source: Original Platform
This commit is contained in:
ModelHub XC
227
README.md
Normal file
227
README.md
Normal file
@@ -0,0 +1,227 @@
|
||||
---
|
||||
license: other
|
||||
license_name: weft-research
|
||||
license_link: https://github.com/wallfacers/data-weave
|
||||
pipeline_tag: text-generation
|
||||
library_name: transformers
|
||||
base_model: Qwen/Qwen2.5-Coder-1.5B-Instruct
|
||||
datasets:
|
||||
- wallfacers/weft-script-lineage-synth
|
||||
language:
|
||||
- en
|
||||
tags:
|
||||
- research-artifact
|
||||
- negative-result
|
||||
- memorization
|
||||
- domain-shift
|
||||
- data-lineage
|
||||
- etl
|
||||
- lora
|
||||
model-index:
|
||||
- name: weft-lineage-extractor-1.5b
|
||||
results:
|
||||
- task:
|
||||
type: table-level-lineage-extraction
|
||||
name: ETL table-level data-lineage extraction
|
||||
dataset:
|
||||
type: synthetic-etl
|
||||
name: synthetic held-out (structural-form isolated)
|
||||
metrics:
|
||||
- type: precision
|
||||
value: 0.995
|
||||
name: Table precision (synthetic held-out)
|
||||
- task:
|
||||
type: table-level-lineage-extraction
|
||||
name: ETL table-level data-lineage extraction
|
||||
dataset:
|
||||
type: real-github-etl
|
||||
name: real GitHub ETL (human gold, n=139)
|
||||
metrics:
|
||||
- type: precision
|
||||
value: 0.270
|
||||
name: Table precision (real, out-of-distribution)
|
||||
- type: accuracy
|
||||
value: 0.496
|
||||
name: Read/write direction accuracy (real)
|
||||
widget:
|
||||
- example_title: Clean literal case (works)
|
||||
text: |
|
||||
task_type: PYTHON
|
||||
script:
|
||||
import psycopg2
|
||||
cur.execute("SELECT id, name FROM users WHERE active = 1")
|
||||
cur.execute("INSERT INTO user_summary (user_id) VALUES (%s)", rows)
|
||||
---
|
||||
|
||||
# weft-lineage-extractor-1.5b
|
||||
|
||||
> ## ⚠️ RESEARCH ARTIFACT — a NEGATIVE RESULT about *synthetic-only* training. Not a production tool.
|
||||
>
|
||||
> ### ✅ Resolved: real-corpus training fixes this. If you want a **usable** lineage extractor, use **[weft-lineage-extractor-3b](https://huggingface.co/wallfacers/weft-lineage-extractor-3b)** — same task, trained on **real** scripts, real precision **0.33 → 0.64**, memorization leak gone.
|
||||
|
||||
A 1.5B model LoRA-fine-tuned **only on synthetic ETL scripts** to extract table-level data
|
||||
lineage. On its **synthetic** held-out set it looks near-perfect (**precision 0.995**). On
|
||||
**real GitHub ETL scripts it collapses** (precision **0.27**), and a large share of its
|
||||
mistakes are **verbatim table names memorized from the synthetic training pool** (**22–40%**
|
||||
of hallucinations, depending on language). It is published so the failure — a systematic
|
||||
pathology of *synthetic-only training* — is reproducible and citable, and so the real-corpus
|
||||
resolution ([3B](https://huggingface.co/wallfacers/weft-lineage-extractor-3b)) has a baseline.
|
||||
|
||||
**Takeaway:** synthetic-benchmark scores for structured-extraction models can be *severely*
|
||||
optimistic. A model can ace a held-out synthetic split by *memorizing the generator's
|
||||
vocabulary*, then emit those memorized names on real, out-of-distribution inputs.
|
||||
|
||||
- **Base:** [Qwen/Qwen2.5-Coder-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct)
|
||||
- **Training data:** 10,000 **synthetic** ETL scripts (Python/Shell, 9 structural forms) — no real scripts in training.
|
||||
- **Companion artifacts:** 0.5B / 3B scale points, a Scala/Java (JVM) variant, and the real-corpus 3B resolution.
|
||||
|
||||
---
|
||||
|
||||
## The headline: synthetic looks great, real does not
|
||||
|
||||
Same model, table-level metrics, identical extraction convention ("Convention A": label a table
|
||||
only if its literal name appears in an executable read/write statement; ignore dynamic names,
|
||||
file paths, temp views, comments, config-driven jobs).
|
||||
|
||||
| Evaluation set | precision | direction acc. | hallucination |
|
||||
|---|---|---|---|
|
||||
| **Synthetic held-out** (600, structural-form isolated) | **0.995** | **0.995** | 0.001 |
|
||||
| **Real GitHub ETL** (139 scripts, human gold) | **0.270** | **0.496** | 0.153 |
|
||||
|
||||
**Four-way comparison on the real Python/Shell set** (n=139, non-empty gold 59):
|
||||
|
||||
| extractor | precision | hallucination | recall (non-∅) | direction (non-∅) |
|
||||
|---|---|---|---|---|
|
||||
| **this model (synthetic 1.5B)** | 0.270 | 0.153 | 0.618 | 0.496 |
|
||||
| Qwen-Max (general LLM) | 0.327 | 0.301 | 0.939 | 0.872 |
|
||||
| Claude (general LLM) | 0.542 | 0.134 | 0.806 | 0.730 |
|
||||
| regex baseline | 0.166 | 0.000 | 0.473 | 0.397 |
|
||||
| **real-corpus 3B (the resolution)** | **0.64** | low | 0.63 | — |
|
||||
|
||||
---
|
||||
|
||||
## Why it fails: memorization leak
|
||||
|
||||
A **hallucination** = a predicted table name that is neither in the gold nor literally present in
|
||||
the script. We check how many are **verbatim** names from the synthetic training pool, or share
|
||||
its **shape** (`schema.schema_base_suffix`, e.g. `dws.dws_member_point_di`).
|
||||
|
||||
| set | hallucinations | verbatim training-pool names | synthetic-shaped |
|
||||
|---|---|---|---|
|
||||
| Python/Shell real | 76 | **17 (22.4%)** | 19 (25.0%) |
|
||||
| JVM (Scala/Java) real | 98 | **40 (40.8%)** | 49 (50.0%) |
|
||||
|
||||
Given a real script it cannot parse, the model **falls back to reciting training table names**.
|
||||
This is the negative result, and it is **gold-independent**.
|
||||
|
||||
### Scale & cross-language
|
||||
|
||||
| scale | synthetic prec | real prec | real direction | **verbatim leak** |
|
||||
|---|---|---|---|---|
|
||||
| 0.5B | 0.994 | 0.243 | 0.369 | **37.4%** |
|
||||
| 1.5B (this) | 0.995 | 0.270 | 0.496 | **22.4%** |
|
||||
| 3B (synthetic) | 0.988 | 0.325 | 0.468 | **10.9%** |
|
||||
| 1.5B + JVM, real JVM eval | ~0.99 | 0.165 | 0.418 | **40.8%** |
|
||||
| **3B, real corpus** | — | **0.64** | — | **~0** |
|
||||
|
||||
Memorization leak shrinks monotonically with model size (a capacity problem), but direction
|
||||
confusion does not improve with scale and the failure reproduces across languages. "More
|
||||
synthetic data" does not close the gap — **real training data does** (bottom row).
|
||||
|
||||
---
|
||||
|
||||
## Intended use
|
||||
|
||||
- ✅ **Reproducing / studying** the synthetic-only-training memorization-leak failure.
|
||||
- ✅ A **baseline** for abstention, real-data augmentation, or leak-mitigation research.
|
||||
- ❌ **Not** for production lineage — use [weft-lineage-extractor-3b](https://huggingface.co/wallfacers/weft-lineage-extractor-3b) instead.
|
||||
|
||||
---
|
||||
|
||||
## Prompt format & quick start
|
||||
|
||||
System prompt (exact — must match training verbatim):
|
||||
|
||||
```
|
||||
You are a data lineage extractor for ETL scripts. Given a PYTHON or SHELL task
|
||||
script, output ONLY a JSON object {"reads": [...], "writes": [...]} where each
|
||||
item is {"table": str, "columns": [str] or null}. Rules: include a table only if
|
||||
its literal name appears in the script text; ignore dynamically-built table names,
|
||||
commented-out SQL, and SQL that is merely printed or logged; if nothing is read or
|
||||
written, output {"reads": [], "writes": []}.
|
||||
```
|
||||
|
||||
```python
|
||||
import json, re, torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
MODEL = "wallfacers/weft-lineage-extractor-1.5b"
|
||||
tok = AutoTokenizer.from_pretrained(MODEL)
|
||||
model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16, device_map="auto").eval()
|
||||
|
||||
SYSTEM = ("You are a data lineage extractor for ETL scripts. Given a PYTHON or SHELL task "
|
||||
"script, output ONLY a JSON object {\"reads\": [...], \"writes\": [...]} where each "
|
||||
"item is {\"table\": str, \"columns\": [str] or null}. Rules: include a table only if "
|
||||
"its literal name appears in the script text; ignore dynamically-built table names, "
|
||||
"commented-out SQL, and SQL that is merely printed or logged; if nothing is read or "
|
||||
"written, output {\"reads\": [], \"writes\": []}.")
|
||||
|
||||
def extract(task_type, script, max_new_tokens=256):
|
||||
msgs = [{"role": "system", "content": SYSTEM},
|
||||
{"role": "user", "content": f"task_type: {task_type}\nscript:\n{script}"}]
|
||||
inp = tok.apply_chat_template(msgs, add_generation_prompt=True, tokenize=True,
|
||||
return_dict=True, return_tensors="pt").to(model.device)
|
||||
with torch.no_grad():
|
||||
out = model.generate(**inp, max_new_tokens=max_new_tokens, do_sample=False,
|
||||
pad_token_id=tok.pad_token_id or tok.eos_token_id)
|
||||
raw = tok.decode(out[0][inp["input_ids"].shape[1]:], skip_special_tokens=True).strip()
|
||||
m = re.search(r"\{.*\}", raw, re.DOTALL)
|
||||
return json.loads(m.group(0)) if m else {"reads": [], "writes": []}
|
||||
|
||||
print(extract("PYTHON", 'cur.execute("SELECT * FROM orders WHERE status = \'pending\'")'))
|
||||
# -> {"reads": [{"table": "orders", "columns": null}], "writes": []}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Training
|
||||
|
||||
| Parameter | Value |
|
||||
|---|---|
|
||||
| Base model | Qwen/Qwen2.5-Coder-1.5B-Instruct |
|
||||
| Method | LoRA (r=16, α=32, dropout=0.05; q/k/v/o/gate/up/down_proj) |
|
||||
| Epochs / LR | 2 / 2e-4 cosine, 3% warmup |
|
||||
| Effective batch / max len | 16 (2×8 grad-accum) / 2048 |
|
||||
| Precision / hardware | bfloat16 / single 12 GB GPU |
|
||||
| Training data | 10,000 **synthetic** ETL scripts (9 structural forms) — **zero real scripts** |
|
||||
| Seed | 20260703 (reproducible) |
|
||||
|
||||
---
|
||||
|
||||
## Limitations & honest disclosures
|
||||
|
||||
- **Not a production tool.** Real-world precision ~0.27; direction ~coin-flip. Use the 3B real-corpus model.
|
||||
- **Literal-only by design:** dynamic names, commented/logged SQL, temp views, config-driven jobs are out of scope.
|
||||
- **Evaluation gold** is human-adjudicated under Convention A; real sets are small (Python/Shell n=139; JVM n=141). The **leak metric is gold-independent** (verbatim 40.4%→40.8% on JVM under full re-adjudication).
|
||||
- **Column-level** output exists in the schema but is best-effort; evaluated claims are table-level.
|
||||
|
||||
---
|
||||
|
||||
## Links & citation
|
||||
|
||||
- **Real-corpus resolution:** [weft-lineage-extractor-3b](https://huggingface.co/wallfacers/weft-lineage-extractor-3b)
|
||||
- **Dataset (synthetic + eval/leak reports):** [wallfacers/weft-script-lineage-synth](https://huggingface.co/datasets/wallfacers/weft-script-lineage-synth)
|
||||
- **Platform:** [Weft (data-weave)](https://github.com/wallfacers/data-weave)
|
||||
|
||||
```bibtex
|
||||
@misc{weft-lineage-negresult-2026,
|
||||
author = {{Weft Contributors}},
|
||||
title = {{Synthetic-only training induces memorization leak in small
|
||||
models for ETL data-lineage extraction: a negative result}},
|
||||
year = 2026,
|
||||
publisher = {{Hugging Face}},
|
||||
howpublished = {{\url{https://huggingface.co/wallfacers/weft-lineage-extractor-1.5b}}},
|
||||
}
|
||||
```
|
||||
|
||||
Trained with [TRL](https://huggingface.co/docs/trl) + [PEFT](https://huggingface.co/docs/peft).
|
||||
Reference in New Issue
Block a user