108 lines
4.1 KiB
Markdown
108 lines
4.1 KiB
Markdown
|
---
|
|||
|
|
license: apache-2.0
|
|||
|
|
base_model: HuggingFaceTB/SmolLM2-135M-Instruct
|
|||
|
|
datasets:
|
|||
|
|
- RomanTeucher/text2cypher-curated
|
|||
|
|
language:
|
|||
|
|
- en
|
|||
|
|
library_name: transformers
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
tags:
|
|||
|
|
- text2cypher
|
|||
|
|
- cypher
|
|||
|
|
- neo4j
|
|||
|
|
- lora
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# SmolLM2-135M-text2cypher
|
|||
|
|
|
|||
|
|
A fine-tune of [`HuggingFaceTB/SmolLM2-135M-Instruct`](https://huggingface.co/HuggingFaceTB/SmolLM2-135M-Instruct)
|
|||
|
|
that turns a natural-language question + a graph schema into a **Cypher** query.
|
|||
|
|
Trained with LoRA on [`RomanTeucher/text2cypher-curated`](https://huggingface.co/datasets/RomanTeucher/text2cypher-curated)
|
|||
|
|
(1000 train / 75 val / 50 test); the adapters are **merged into the base weights**,
|
|||
|
|
so this is a standalone full model that loads like any HF checkpoint — no PEFT needed.
|
|||
|
|
**This model is a submission for an interview.**
|
|||
|
|
|
|||
|
|
## Usage
|
|||
|
|
|
|||
|
|
The model expects the same ChatML prompt it was trained on: a fixed system message
|
|||
|
|
plus a user turn of `Schema:` + `Question:`.
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|||
|
|
|
|||
|
|
model_id = "rizwan261/smollm2-135m-text2cypher"
|
|||
|
|
tok = AutoTokenizer.from_pretrained(model_id)
|
|||
|
|
model = AutoModelForCausalLM.from_pretrained(model_id)
|
|||
|
|
|
|||
|
|
SYSTEM = ("You are a text-to-Cypher engine for Neo4j. Given a graph schema and a "
|
|||
|
|
"question, output a single valid Cypher query that answers the question. "
|
|||
|
|
"Use only labels, relationship types and properties that appear in the "
|
|||
|
|
"schema. Output the Cypher query and nothing else: no explanation, no "
|
|||
|
|
"comments, no markdown code fences.")
|
|||
|
|
|
|||
|
|
schema = "Graph schema: Relevant node labels and their properties (with datatypes) are:\nUpdateDate {update_date: DATE}"
|
|||
|
|
question = "Which nodes are connected to UpdateDate where update_date is 2008-01-29, and also to another node?"
|
|||
|
|
messages = [
|
|||
|
|
{"role": "system", "content": SYSTEM},
|
|||
|
|
{"role": "user", "content": f"Schema:\n{schema}\n\nQuestion:\n{question}"},
|
|||
|
|
]
|
|||
|
|
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
|
|||
|
|
out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
|
|||
|
|
print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## Training
|
|||
|
|
|
|||
|
|
LoRA (r=16, α=32, dropout=0.05) on the attention + MLP projections, completion-only
|
|||
|
|
loss (the prompt is masked, loss is on the target Cypher only), then merged.
|
|||
|
|
|
|||
|
|
| | |
|
|||
|
|
|---|---|
|
|||
|
|
| epochs | 20 |
|
|||
|
|
| learning rate | 5e-5, cosine, 5% warmup |
|
|||
|
|
| effective batch | 16 (bs 4 × grad-accum 4) |
|
|||
|
|
| max length | 1024 |
|
|||
|
|
| best val loss | ~0.32 (by `eval_loss`) |
|
|||
|
|
|
|||
|
|
## Evaluation
|
|||
|
|
|
|||
|
|
Test split (n=50), greedy decoding. **Translation / structural metrics** over the
|
|||
|
|
whole split — surface proxies for correctness:
|
|||
|
|
|
|||
|
|
| metric | base | this model |
|
|||
|
|
| --- | ---: | ---: |
|
|||
|
|
| exact_match | 0.00 | 0.26 |
|
|||
|
|
| structural_f1 | 0.18 | 0.81 |
|
|||
|
|
| google_bleu | 0.07 | 0.70 |
|
|||
|
|
| well_formed | 0.00 | 0.98 |
|
|||
|
|
|
|||
|
|
**Execution-based** evaluation against the live Neo4j demo databases, on the 18/50
|
|||
|
|
samples that carry a database reference — predicted and gold queries are run and
|
|||
|
|
their *result sets* compared. This is the honest measure, and each stricter layer
|
|||
|
|
peels back the previous one's optimism:
|
|||
|
|
|
|||
|
|
| how strict | metric | base | this model |
|
|||
|
|
| --- | --- | ---: | ---: |
|
|||
|
|
| real parser + live schema | CyVer KG-Valid-Query | 0.00 | 0.61 |
|
|||
|
|
| runs on the DB | query executes | 0.00 | 0.72 |
|
|||
|
|
| **returns the right rows** | exec ExactMatch (non-trivial) | 0.00 | **0.00** |
|
|||
|
|
|
|||
|
|
The surface metrics jump, but execution accuracy stays at zero: the model writes
|
|||
|
|
well-formed, schema-valid, runnable Cypher that returns the *wrong data*.
|
|||
|
|
|
|||
|
|
## Limitations
|
|||
|
|
|
|||
|
|
A 135M model produces Cypher that often *looks* right but isn't: only ~26% are
|
|||
|
|
exact matches, and **non-trivial execution accuracy is ~0** — well-formed,
|
|||
|
|
runnable queries that return the wrong rows. It reliably gets node labels, clauses
|
|||
|
|
and the query skeleton, but struggles with same-node/co-reference patterns,
|
|||
|
|
multi-stage aggregation, and occasionally hallucinates functions. Execution
|
|||
|
|
numbers come from a small (n=18) subset run against live demo databases, so they
|
|||
|
|
can drift.
|
|||
|
|
|
|||
|
|
## License
|
|||
|
|
|
|||
|
|
Apache-2.0, inherited from the SmolLM2 base model.
|