108 lines
4.1 KiB
Markdown
108 lines
4.1 KiB
Markdown
---
|
||
license: apache-2.0
|
||
base_model: HuggingFaceTB/SmolLM2-135M-Instruct
|
||
datasets:
|
||
- RomanTeucher/text2cypher-curated
|
||
language:
|
||
- en
|
||
library_name: transformers
|
||
pipeline_tag: text-generation
|
||
tags:
|
||
- text2cypher
|
||
- cypher
|
||
- neo4j
|
||
- lora
|
||
---
|
||
|
||
# SmolLM2-135M-text2cypher
|
||
|
||
A fine-tune of [`HuggingFaceTB/SmolLM2-135M-Instruct`](https://huggingface.co/HuggingFaceTB/SmolLM2-135M-Instruct)
|
||
that turns a natural-language question + a graph schema into a **Cypher** query.
|
||
Trained with LoRA on [`RomanTeucher/text2cypher-curated`](https://huggingface.co/datasets/RomanTeucher/text2cypher-curated)
|
||
(1000 train / 75 val / 50 test); the adapters are **merged into the base weights**,
|
||
so this is a standalone full model that loads like any HF checkpoint — no PEFT needed.
|
||
**This model is a submission for an interview.**
|
||
|
||
## Usage
|
||
|
||
The model expects the same ChatML prompt it was trained on: a fixed system message
|
||
plus a user turn of `Schema:` + `Question:`.
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
||
model_id = "rizwan261/smollm2-135m-text2cypher"
|
||
tok = AutoTokenizer.from_pretrained(model_id)
|
||
model = AutoModelForCausalLM.from_pretrained(model_id)
|
||
|
||
SYSTEM = ("You are a text-to-Cypher engine for Neo4j. Given a graph schema and a "
|
||
"question, output a single valid Cypher query that answers the question. "
|
||
"Use only labels, relationship types and properties that appear in the "
|
||
"schema. Output the Cypher query and nothing else: no explanation, no "
|
||
"comments, no markdown code fences.")
|
||
|
||
schema = "Graph schema: Relevant node labels and their properties (with datatypes) are:\nUpdateDate {update_date: DATE}"
|
||
question = "Which nodes are connected to UpdateDate where update_date is 2008-01-29, and also to another node?"
|
||
messages = [
|
||
{"role": "system", "content": SYSTEM},
|
||
{"role": "user", "content": f"Schema:\n{schema}\n\nQuestion:\n{question}"},
|
||
]
|
||
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
|
||
out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
|
||
print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
|
||
|
||
```
|
||
|
||
## Training
|
||
|
||
LoRA (r=16, α=32, dropout=0.05) on the attention + MLP projections, completion-only
|
||
loss (the prompt is masked, loss is on the target Cypher only), then merged.
|
||
|
||
| | |
|
||
|---|---|
|
||
| epochs | 20 |
|
||
| learning rate | 5e-5, cosine, 5% warmup |
|
||
| effective batch | 16 (bs 4 × grad-accum 4) |
|
||
| max length | 1024 |
|
||
| best val loss | ~0.32 (by `eval_loss`) |
|
||
|
||
## Evaluation
|
||
|
||
Test split (n=50), greedy decoding. **Translation / structural metrics** over the
|
||
whole split — surface proxies for correctness:
|
||
|
||
| metric | base | this model |
|
||
| --- | ---: | ---: |
|
||
| exact_match | 0.00 | 0.26 |
|
||
| structural_f1 | 0.18 | 0.81 |
|
||
| google_bleu | 0.07 | 0.70 |
|
||
| well_formed | 0.00 | 0.98 |
|
||
|
||
**Execution-based** evaluation against the live Neo4j demo databases, on the 18/50
|
||
samples that carry a database reference — predicted and gold queries are run and
|
||
their *result sets* compared. This is the honest measure, and each stricter layer
|
||
peels back the previous one's optimism:
|
||
|
||
| how strict | metric | base | this model |
|
||
| --- | --- | ---: | ---: |
|
||
| real parser + live schema | CyVer KG-Valid-Query | 0.00 | 0.61 |
|
||
| runs on the DB | query executes | 0.00 | 0.72 |
|
||
| **returns the right rows** | exec ExactMatch (non-trivial) | 0.00 | **0.00** |
|
||
|
||
The surface metrics jump, but execution accuracy stays at zero: the model writes
|
||
well-formed, schema-valid, runnable Cypher that returns the *wrong data*.
|
||
|
||
## Limitations
|
||
|
||
A 135M model produces Cypher that often *looks* right but isn't: only ~26% are
|
||
exact matches, and **non-trivial execution accuracy is ~0** — well-formed,
|
||
runnable queries that return the wrong rows. It reliably gets node labels, clauses
|
||
and the query skeleton, but struggles with same-node/co-reference patterns,
|
||
multi-stage aggregation, and occasionally hallucinates functions. Execution
|
||
numbers come from a small (n=18) subset run against live demo databases, so they
|
||
can drift.
|
||
|
||
## License
|
||
|
||
Apache-2.0, inherited from the SmolLM2 base model.
|