qwen2.5-3b-text2sql/README.md

---
base_model: unsloth/Qwen2.5-3B-Instruct
library_name: transformers
pipeline_tag: text-generation
license: apache-2.0
language:
- en
tags:
- text-to-sql
- sql
- qwen2
- lora
- qlora
- unsloth
- trl
datasets:
- b-mc2/sql-create-context
---

# Qwen2.5-3B Text-to-SQL

A fine-tuned version of **Qwen2.5-3B-Instruct** that converts natural-language
questions into SQL queries, given a database schema. Trained with LoRA/QLoRA on a
single free GPU.

- **Developed by:** ashishsahu2008
- **Base model:** [unsloth/Qwen2.5-3B-Instruct](https://huggingface.co/unsloth/Qwen2.5-3B-Instruct)
- **Fine-tuned on:** [b-mc2/sql-create-context](https://huggingface.co/datasets/b-mc2/sql-create-context)
- **License:** apache-2.0
- **Live demo:** https://huggingface.co/spaces/ashishsahu2008/text2sql-demo
- **Code / write-up:** https://github.com/ashishsahu2008/fine-tuning

## What it does

Give it a `CREATE TABLE` schema and a plain-English question, and it returns only
the SQL query that answers the question — no explanations or markdown, just runnable SQL.

## Results

Evaluated on a held-out test split (n=200) the model never saw during training.
Both the base and fine-tuned models were scored with identical string normalization
(lowercase, strip `;`, canonicalize quotes), so the comparison is apples-to-apples.

| Model                          | Exact-match accuracy |
|--------------------------------|:--------------------:|
| Qwen2.5-3B-Instruct (base)     |        41.5%         |
| **This model (fine-tuned)**    |      **72.5%**       |

Only 0.96% of parameters (~30M of 3.1B) were trained via LoRA. Much of the gain
comes from the base model learning to emit clean, executable SQL instead of wrapping
answers in prose or markdown fences.

> **Note on the metric:** exact-match undercounts semantically-correct queries that
> are written differently (column order, aliases, whitespace). A stronger evaluation
> would use AST comparison (`sqlglot`) or execution accuracy on a benchmark with
> populated databases (Spider / BIRD); `sql-create-context` ships schemas only.

## Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "ashishsahu2008/qwen2.5-3b-text2sql"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16)

schema = "CREATE TABLE employees (name VARCHAR, salary INTEGER, department VARCHAR)"
question = "List the names of employees in Sales earning over 50000."

messages = [
    {"role": "system", "content": "You are a SQL expert. Given a database schema "
                                   "and a question, output only the SQL query that answers it."},
    {"role": "user", "content": f"Schema:\n{schema}\n\nQuestion: {question}"},
]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)
out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```

## Prompt format

The model expects the schema and question in a single user turn, with this system prompt:

```
System:  You are a SQL expert. Given a database schema and a question, output only the SQL query that answers it.
User:    Schema:
         <CREATE TABLE ...>

         Question: <your question>
```

## Training details

- **Method:** supervised fine-tuning (SFT) with LoRA on a 4-bit quantized base (QLoRA)
- **LoRA config:** rank 16, alpha 16, applied to attention + MLP projections
- **Data:** 3,000-row subset of `b-mc2/sql-create-context` (2,700 train / 300 test)
- **Hyperparameters:** 2 epochs, learning rate 2e-4, effective batch size 8
- **Hardware:** single Colab T4, ~25 minutes
- **Frameworks:** [Unsloth](https://github.com/unslothai/unsloth) + TRL `SFTTrainer`

## Limitations

- Trained on **single-table** `CREATE TABLE` schemas; complex multi-join databases
  are out of distribution.
- Assumes the schema is provided in the prompt — it does not know any tables you
  don't give it.
- SQL dialect follows the training data (broadly SQLite-compatible); it does not
  target a specific engine's extensions.

Fine-tuned with [Unsloth](https://github.com/unslothai/unsloth) and Hugging Face's TRL library.