weft-lineage-extractor-jvm-1.5b/README.md

license, license_name, license_link, pipeline_tag, library_name, base_model, datasets, language, tags, model-index

license	license_name	license_link	pipeline_tag	library_name	base_model	datasets	language	tags	model-index
other	weft-research	https://github.com/wallfacers/data-weave	text-generation	transformers	Qwen/Qwen2.5-Coder-1.5B-Instruct	wallfacers/weft-script-lineage-synth	en	research-artifact negative-result memorization domain-shift cross-language data-lineage etl lora	name results weft-lineage-extractor-jvm-1.5b task dataset metrics type name table-level-lineage-extraction ETL table-level data-lineage extraction (Scala/Java) type name real-jvm-etl real Scala/Java ETL (human gold, n=141) type value name precision 0.165 Table precision (real JVM, out-of-distribution) type value name accuracy 0.418 Read/write direction accuracy (real JVM)

weft-lineage-extractor-jvm-1.5b — cross-language reproduction of a NEGATIVE RESULT

⚠️ RESEARCH ARTIFACT — the Scala/Java (JVM) point of a synthetic-only training study. Not a production tool.

✅ Resolved by real data: use weft-lineage-extractor-3b (real corpus, real precision 0.64). Full study: weft-lineage-extractor-1.5b.

The JVM (Scala/Java) point of a study showing that synthetic-only training induces a verbatim memorization leak in small ETL table-lineage models — and that the failure reproduces, worse, across languages. Synthetic held-out looks near-perfect (~0.99); on real Scala/Java ETL it drops to precision 0.165, with 40.8% of hallucinations being table names recited verbatim from the synthetic training pool — the highest leak of all scale/language points, because real JVM code is the most out-of-distribution.

Same recipe as the 1.5B main model (LoRA on Qwen2.5-Coder-1.5B), extended with synthetic Scala/Java forms; task_type: SCALA | JAVA.

This point's numbers (table-level, Convention A)

metric	synthetic held-out	real Scala/Java ETL (n=141, non-empty 28)
precision	~0.99	0.165
direction accuracy	—	0.418
verbatim memorization leak	—	40.8%

The leak metric is gold-independent and stable to full-file re-adjudication (verbatim 40.4% → 40.8%). Given a real SqlDao.java it cannot parse, the model emits training-pool names like dws.dws_member_point_df — memorized generator vocabulary, not a label artifact.

Where it fits (scale / cross-language curve)

point	real precision	real direction	verbatim leak
0.5B (Py/Sh)	0.243	0.369	37.4%
1.5B (Py/Sh)	0.270	0.496	22.4%
3B (Py/Sh, synthetic)	0.325	0.468	10.9%
1.5B + JVM (this, real JVM)	0.165	0.418	40.8%
3B (real corpus)	0.64	—	~0

"Add another language" does not fix the real-world gap; real training data does.

Intended use

• ✅ Reproducing / studying the cross-language reproduction of the memorization-leak failure.
• ❌ Not for production lineage — use the real-corpus 3B.

Usage, prompt format, training details, citation

Identical to the main model (this variant adds task_type: SCALA | JAVA and JVM system-prompt wording): weft-lineage-extractor-1.5b.

• Dataset & reports: wallfacers/weft-script-lineage-synth
• Base model: Qwen/Qwen2.5-Coder-1.5B-Instruct
• Platform: Weft (data-weave)