qwen2.5-1.5b-sql-qlora-merged/README.md

---
license: apache-2.0
base_model: Qwen/Qwen2.5-1.5B-Instruct
library_name: transformers
tags:
- text-to-sql
- sql
- qlora
- lora
- peft
- qwen2.5
datasets:
- b-mc2/sql-create-context
language:
- en
pipeline_tag: text-generation
model-index:
- name: qwen2.5-1.5b-sql-qlora-merged
  results:
  - task:
      type: text2text-generation
      name: Text-to-SQL
    dataset:
      name: sql-create-context
      type: b-mc2/sql-create-context
      split: held-out test (1000 examples)
    metrics:
    - type: exact_match
      value: 0.748
      name: Normalized exact match
    - type: f1
      value: 0.9732
      name: SQL token F1
---

# qwen2.5-1.5b-sql-qlora-merged

Merged fp16 weights of [`Qwen/Qwen2.5-1.5B-Instruct`](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct),
QLoRA-fine-tuned for **text-to-SQL**: given a `CREATE TABLE` schema and a natural-language
question, emit a single SQLite query.

The LoRA adapter alone is at
[`blaze-star/qwen2.5-1.5b-sql-qlora`](https://huggingface.co/blaze-star/qwen2.5-1.5b-sql-qlora).

## Results

Held-out test split, 1,000 examples never seen in training.

| Model | Exact match | Token F1 | Format compliance |
|---|---|---|---|
| Base, 0-shot (4-bit) | 49.7% | 0.925 | 26.7% |
| Base, 3-shot (4-bit) | 52.3% | 0.921 | 99.3% |
| **QLoRA fine-tuned** | **74.8%** | **0.973** | **99.9%** |
| **Delta vs 0-shot** | **+25.1 pts** | **+0.049** | **+73.2 pts** |

Fine-tuning improved exact match by **+25.1 points** (49.7% -> 74.8%), a +51% relative gain.

## Usage

The model expects the chat template with this system prompt — it was trained with it, and
accuracy drops without it.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("blaze-star/qwen2.5-1.5b-sql-qlora-merged", dtype=torch.float16, device_map="auto")
tok = AutoTokenizer.from_pretrained("blaze-star/qwen2.5-1.5b-sql-qlora-merged")
```

```python
SYSTEM = ("You are a text-to-SQL engine. Given a SQLite schema and a question, reply with a "
          "single SQL query that answers the question. Output only the SQL query: no "
          "explanation, no comments, no markdown code fences.")

schema = "CREATE TABLE head (age INTEGER)"
question = "How many heads of the departments are older than 56?"

messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": f"Schema:\n{schema}\n\nQuestion: {question}\n\nSQL:"},
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
ids = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=96, do_sample=False)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))
# SELECT COUNT(*) FROM head WHERE age > 56
```

Use greedy decoding (`do_sample=False`). Sampling hurts exact match on this task.

## What the metric actually measures

Primary metric is **normalized exact match** against the dataset's reference SQL:
lowercased, whitespace collapsed, `"` unified to `'`, backticks/brackets stripped,
spacing normalized around operators and punctuation. It is strict — a semantically
equivalent query that differs in alias naming (`AS p` vs `AS T1`) or literal quoting
(`= '15'` vs `= 15`) counts as a miss.

That strictness is the point, but it must be read correctly: **the base model already
produces largely correct SQL content** (token F1 0.925 before any training).
Much of the headroom is conformance to this dataset's canonical SQL style, which is
exactly what task-specific fine-tuning buys you. To keep that claim honest this project
reports three separate baselines rather than one:

| Baseline | Exact match | Format compliance | What it controls for |
|---|---|---|---|
| 4-bit, 0-shot | 49.7% | 26.7% | matched conditions — same quantization the adapter trains on |
| 4-bit, 3-shot | 52.3% | 99.3% | isolates *output formatting* from *SQL convention* |
| fp16, 0-shot | 57.3% | 99.4% | strongest untrained configuration |

The 3-shot baseline is the important control. Three in-context examples raise format
compliance to 99.3% — the model stops wrapping output in markdown fences almost
entirely — yet exact match moves only to 52.3%. **Formatting was therefore not the
bottleneck**, and gains above that line are genuine SQL-convention learning, not
prompt-format cleanup.

Both models receive an identical prompt and identical output post-processing
(fence stripping, leading-prose removal, first-statement extraction), so neither is
advantaged by the harness. Secondary metrics: order-insensitive **token F1** over SQL
tokens (partial credit) and **format compliance** (fraction of raw generations that were
already bare SQL).

## Data and leakage control

[`b-mc2/sql-create-context`](https://huggingface.co/datasets/b-mc2/sql-create-context) — natural-language question +
`CREATE TABLE` schema -> SQLite query.

The 78,577 raw rows are deduplicated on a SHA-1 of the normalized
`(question, schema)` pair (4 exact duplicates dropped), shuffled with seed
42, and then **test is carved off first**, before val and train. Splits:
**12,000 train / 750 val / 1,000 test**.

Split disjointness is asserted at build time and recorded in
[`data/split_report.json`](https://github.com/harshb20/qwen2.5-1.5b-sql-qlora/blob/main/data/split_report.json):

```json
{"train_test_overlap": 0, "val_test_overlap": 0, "train_val_overlap": 0}
```

`prepare_data.py` raises if any of these is non-zero, so a leaking split cannot be
trained on. The test split was used only by `evaluate.py`, never by `train.py`.

## Training

| | |
|---|---|
| Base model | [`Qwen/Qwen2.5-1.5B-Instruct`](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct) |
| Method | QLoRA — frozen 4-bit NF4 base (double quant, bf16 compute) + LoRA adapters |
| LoRA | r=16, alpha=32, dropout=0.05, on `q,k,v,o,gate,up,down_proj` |
| Trainable params | 18.5M of 1.56B (1.18%) |
| Optimizer | paged AdamW 8-bit, lr 0.0002, cosine schedule, 3% warmup, grad-clip 0.3 |
| Effective batch | 32 (16 x 2 accumulation) |
| Epochs | 2 |
| Max seq len | 512 tokens (observed mean 125, max 254) |
| Loss | cross-entropy on the **SQL completion only** — prompt tokens masked to -100 |
| Hardware | 1x NVIDIA A100-SXM4-80GB |
| Wall time | 11.7 min |
| Peak VRAM (training) | 30.96 GB |

Loss is computed only on the assistant turn, so the model is never rewarded for
reproducing the schema or the question.

![loss curves](loss_curves.png)

## Quantization: latency, VRAM, and quality

Merged fp16 model re-quantized with bitsandbytes and benchmarked on the same A100.
Latency is a single request generating exactly 64 tokens (20 runs after 3 warmups);
quality is exact match on the first 300 test examples.

| Precision | Weights VRAM | Peak VRAM | Latency (bs=1, 64 tok) | Decode tok/s | Batch-16 tok/s | Exact match |
|---|---|---|---|---|---|---|
| fp16 | 3.09 GB | 3.3 GB | 1862.4 ms | 34.4 | 436.5 | 76.7% (n=300) |
| 8bit | 1.8 GB | 2.1 GB | 27963.2 ms | 2.3 | 16.7 | 75.3% (n=300) |
| 4bit | 1.16 GB | 1.54 GB | 2320.7 ms | 27.6 | 369.2 | 75.3% (n=300) |

## Limitations

- **Single-table, synthetic-ish schemas.** `sql-create-context` schemas are small
  `CREATE TABLE` statements derived from WikiSQL/Spider. Performance will not transfer
  directly to large multi-table production warehouses.
- **Exact match is style-sensitive.** A correct query written in a different but valid
  style scores zero. Token F1 is reported alongside for this reason.
- **No execution-based evaluation.** Queries are compared as strings, not run against a
  database, so semantic equivalence is undercounted.
- **4-bit inference costs accuracy.** The base model loses ~8 points of exact match
  going from fp16 to 4-bit (57.3% -> 49.7%); see the quantization table for the
  fine-tuned model's own fp16/8-bit/4-bit spread.
- **English only**, and the model emits SQLite dialect.


## Intended use

Converting natural-language questions into SQLite queries over small, explicitly-provided
schemas — a component inside a larger system that supplies the schema and validates or
sandboxes the generated query. **Do not execute generated SQL against a production
database without validation**; the model can emit syntactically valid queries that are
semantically wrong.

## Training code

Full, reproducible pipeline: [https://github.com/harshb20/qwen2.5-1.5b-sql-qlora](https://github.com/harshb20/qwen2.5-1.5b-sql-qlora)

## Citation

```bibtex
@misc{qwen25-1.5b-sql-qlora,
  title  = {QLoRA text-to-SQL fine-tune of Qwen2.5-1.5B-Instruct},
  author = {harshb20},
  year   = {2026},
  url    = {https://github.com/harshb20/qwen2.5-1.5b-sql-qlora}
}
```