92 lines
3.8 KiB
Markdown
92 lines
3.8 KiB
Markdown
|
---
|
||
|
|
license: other
|
||
|
|
license_name: weft-research
|
||
|
|
license_link: https://github.com/wallfacers/data-weave
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
library_name: transformers
|
||
|
|
base_model: Qwen/Qwen2.5-Coder-1.5B-Instruct
|
||
|
|
datasets:
|
||
|
|
- wallfacers/weft-script-lineage-synth
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
tags:
|
||
|
|
- research-artifact
|
||
|
|
- negative-result
|
||
|
|
- memorization
|
||
|
|
- domain-shift
|
||
|
|
- cross-language
|
||
|
|
- data-lineage
|
||
|
|
- etl
|
||
|
|
- lora
|
||
|
|
model-index:
|
||
|
|
- name: weft-lineage-extractor-jvm-1.5b
|
||
|
|
results:
|
||
|
|
- task:
|
||
|
|
type: table-level-lineage-extraction
|
||
|
|
name: ETL table-level data-lineage extraction (Scala/Java)
|
||
|
|
dataset:
|
||
|
|
type: real-jvm-etl
|
||
|
|
name: real Scala/Java ETL (human gold, n=141)
|
||
|
|
metrics:
|
||
|
|
- type: precision
|
||
|
|
value: 0.165
|
||
|
|
name: Table precision (real JVM, out-of-distribution)
|
||
|
|
- type: accuracy
|
||
|
|
value: 0.418
|
||
|
|
name: Read/write direction accuracy (real JVM)
|
||
|
|
---
|
||
|
|
|
||
|
|
# weft-lineage-extractor-jvm-1.5b — cross-language reproduction of a NEGATIVE RESULT
|
||
|
|
|
||
|
|
> ## ⚠️ RESEARCH ARTIFACT — the Scala/Java (JVM) point of a *synthetic-only* training study. Not a production tool.
|
||
|
|
>
|
||
|
|
> ### ✅ Resolved by real data: use **[weft-lineage-extractor-3b](https://huggingface.co/wallfacers/weft-lineage-extractor-3b)** (real corpus, real precision **0.64**). Full study: **[weft-lineage-extractor-1.5b](https://huggingface.co/wallfacers/weft-lineage-extractor-1.5b)**.
|
||
|
|
|
||
|
|
The **JVM (Scala/Java)** point of a study showing that **synthetic-only training** induces a
|
||
|
|
**verbatim memorization leak** in small ETL table-lineage models — and that the failure
|
||
|
|
**reproduces, worse, across languages**. Synthetic held-out looks near-perfect (~0.99); on real
|
||
|
|
Scala/Java ETL it drops to **precision 0.165**, with **40.8%** of hallucinations being table names
|
||
|
|
recited verbatim from the synthetic training pool — the **highest leak** of all scale/language
|
||
|
|
points, because real JVM code is the most out-of-distribution.
|
||
|
|
|
||
|
|
Same recipe as the [1.5B main model](https://huggingface.co/wallfacers/weft-lineage-extractor-1.5b)
|
||
|
|
(LoRA on Qwen2.5-Coder-1.5B), extended with synthetic Scala/Java forms; `task_type: SCALA | JAVA`.
|
||
|
|
|
||
|
|
## This point's numbers (table-level, Convention A)
|
||
|
|
|
||
|
|
| metric | synthetic held-out | real Scala/Java ETL (n=141, non-empty 28) |
|
||
|
|
|---|---|---|
|
||
|
|
| precision | ~0.99 | **0.165** |
|
||
|
|
| direction accuracy | — | **0.418** |
|
||
|
|
| **verbatim memorization leak** | — | **40.8%** |
|
||
|
|
|
||
|
|
The leak metric is **gold-independent** and stable to full-file re-adjudication (verbatim
|
||
|
|
40.4% → 40.8%). Given a real `SqlDao.java` it cannot parse, the model emits training-pool names
|
||
|
|
like `dws.dws_member_point_df` — memorized generator vocabulary, not a label artifact.
|
||
|
|
|
||
|
|
## Where it fits (scale / cross-language curve)
|
||
|
|
|
||
|
|
| point | real precision | real direction | verbatim leak |
|
||
|
|
|---|---|---|---|
|
||
|
|
| 0.5B (Py/Sh) | 0.243 | 0.369 | 37.4% |
|
||
|
|
| 1.5B (Py/Sh) | 0.270 | 0.496 | 22.4% |
|
||
|
|
| 3B (Py/Sh, synthetic) | 0.325 | 0.468 | 10.9% |
|
||
|
|
| **1.5B + JVM (this, real JVM)** | **0.165** | **0.418** | **40.8%** |
|
||
|
|
| **3B (real corpus)** | **0.64** | — | **~0** |
|
||
|
|
|
||
|
|
"Add another language" does not fix the real-world gap; **real training data does**.
|
||
|
|
|
||
|
|
## Intended use
|
||
|
|
|
||
|
|
- ✅ Reproducing / studying the cross-language reproduction of the memorization-leak failure.
|
||
|
|
- ❌ Not for production lineage — use the [real-corpus 3B](https://huggingface.co/wallfacers/weft-lineage-extractor-3b).
|
||
|
|
|
||
|
|
## Usage, prompt format, training details, citation
|
||
|
|
|
||
|
|
Identical to the main model (this variant adds `task_type: SCALA | JAVA` and JVM system-prompt wording):
|
||
|
|
**[weft-lineage-extractor-1.5b](https://huggingface.co/wallfacers/weft-lineage-extractor-1.5b)**.
|
||
|
|
|
||
|
|
- **Dataset & reports:** [wallfacers/weft-script-lineage-synth](https://huggingface.co/datasets/wallfacers/weft-script-lineage-synth)
|
||
|
|
- **Base model:** [Qwen/Qwen2.5-Coder-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct)
|
||
|
|
- **Platform:** [Weft (data-weave)](https://github.com/wallfacers/data-weave)
|