llama-3.2-3b-text2sql/README.md

---
base_model: unsloth/Llama-3.2-3B-Instruct-bnb-4bit
tags:
  - text-generation-inference
  - transformers
  - unsloth
  - llama
  - text-to-sql
  - peft
  - lora
license: apache-2.0
language:
  - en
datasets:
  - gretelai/synthetic_text_to_sql
---

# 🔮 Llama-3.2-3B-Instruct Text-to-SQL

A fine-tuned version of [`unsloth/Llama-3.2-3B-Instruct-bnb-4bit`](https://huggingface.co/unsloth/Llama-3.2-3B-Instruct-bnb-4bit) for generating SQL queries from natural language questions and database DDL schemas.

[![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![Finetuned with Unsloth](https://img.shields.io/badge/Finetuned%20with-Unsloth-blue.svg)](https://github.com/unslothai/unsloth)

---

## Model Summary

| Attribute | Value |
|:---|:---|
| **Base Model** | `unsloth/Llama-3.2-3B-Instruct-bnb-4bit` |
| **Task** | Text-to-SQL (natural language → SQL) |
| **Fine-Tuning Method** | LoRA (PEFT) via Unsloth + TRL |
| **Training Dataset** | `gretelai/synthetic_text_to_sql` (50k samples) |
| **Trainable Parameters** | ~0.75% of base model |
| **Export Format** | Merged FP16 (`merged_16bit`) |
| **License** | Apache-2.0 + Meta Llama 3 Community License |
| **Developer** | A-Kishore |

---

## Evaluation Results

Evaluated on the first 200 samples of the `gretelai/synthetic_text_to_sql` test split using greedy decoding. ROUGE F-measures reported.

| Model | ROUGE-1 | ROUGE-2 | ROUGE-L |
|:---|:---:|:---:|:---:|
| Base Model (`unsloth/Llama-3.2-3B-Instruct-bnb-4bit`) | 0.2908 | 0.2016 | 0.2651 |
| Fine-Tuned (`A-Kishore/llama-3.2-3b-text2sql`) | **0.8486** | **0.7232** | **0.8151** |
| **Improvement** | **+191.82%** | **+258.73%** | **+207.47%** |

**Metric interpretation:**
- **ROUGE-1** (unigram overlap) reflects accurate retrieval of schema identifiers and SQL keywords.
- **ROUGE-2** (bigram overlap) captures structural alignment of consecutive SQL constructs (e.g. `GROUP BY`, `ORDER BY`).
- **ROUGE-L** (longest common subsequence) tracks overall query flow including nested clauses and join ordering.

> **Note:** ROUGE measures lexical overlap, not SQL executability. A query may score slightly lower due to stylistic differences (alias names, join ordering) while still being functionally equivalent. See [Limitations](#limitations).

---

## How to Use

The model weights are fully merged in 16-bit precision and load with standard `transformers` or `unsloth`.

### Prompt Format

Always use this exact template — the model was trained on it:

```
###TASK
Generate the SQL query to answer the following question

### Database Schema
{sql_context}

### Question
{sql_prompt}

### SQL Query
```

### (a) Standard `transformers`

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "A-Kishore/llama-3.2-3b-text2sql"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    device_map="auto"
)

prompt = """###TASK
Generate the SQL query to answer the following question

### Database Schema
{sql_context}

### Question
{sql_prompt}

### SQL Query
"""

sql_context = "CREATE TABLE employees (id INT, name TEXT, department TEXT, salary REAL);"
sql_prompt = "What is the average salary per department?"

inputs = tokenizer(
    prompt.format(sql_context=sql_context, sql_prompt=sql_prompt),
    return_tensors="pt"
).to("cuda")

outputs = model.generate(
    **inputs,
    max_new_tokens=150,
    use_cache=True,
    pad_token_id=tokenizer.eos_token_id
)

result = tokenizer.decode(outputs[0], skip_special_tokens=True)
sql = result.split("### SQL Query")[-1].strip()
print(sql)
# SELECT department, AVG(salary) FROM employees GROUP BY department;
```

### (b) Unsloth Fast Inference

```python
import torch
from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="A-Kishore/llama-3.2-3b-text2sql",
    max_seq_length=768,
    dtype=torch.float16,
    load_in_4bit=False,
)
FastLanguageModel.for_inference(model)

prompt = """###TASK
Generate the SQL query to answer the following question

### Database Schema
{sql_context}

### Question
{sql_prompt}

### SQL Query
"""

sql_context = "CREATE TABLE employees (id INT, name TEXT, department TEXT, salary REAL);"
sql_prompt = "What is the average salary per department?"

inputs = tokenizer(
    prompt.format(sql_context=sql_context, sql_prompt=sql_prompt),
    return_tensors="pt"
).to("cuda")

outputs = model.generate(
    **inputs,
    max_new_tokens=150,
    temperature=None,
    do_sample=False,
    pad_token_id=tokenizer.eos_token_id
)

result = tokenizer.decode(outputs[0], skip_special_tokens=True)
sql = result.split("### SQL Query")[-1].strip()
print(sql)
```

---

## Training Details

### Dataset

- **Source:** [`gretelai/synthetic_text_to_sql`](https://huggingface.co/datasets/gretelai/synthetic_text_to_sql)
- **Training split:** 50,000 shuffled samples from `train`
- **Evaluation split:** First 200 samples from `test`

### LoRA Configuration

| Parameter | Value |
|:---|:---|
| Rank (`r`) | `16` |
| Alpha (`lora_alpha`) | `16` |
| Target Modules | `q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj` |
| Bias | `none` |

LoRA freezes the base model weights and injects trainable rank-decomposition matrices into all attention and MLP projections. Only ~0.75% of parameters are updated, dramatically reducing VRAM usage and preventing catastrophic forgetting.

### Hyperparameters

| Parameter | Value |
|:---|:---|
| Optimizer | `paged_adamw_8bit` |
| Learning Rate | `2e-4` |
| LR Scheduler | `linear` |
| Warmup Steps | `5` |
| Epochs | `1` |
| Per-Device Batch Size | `8` |
| Gradient Accumulation | `1` |
| Max Sequence Length | `768` |
| Sequence Packing | `True` |
| Mixed Precision | `fp16` |
| Experiment Tracking | Weights & Biases |

Training was accelerated using the `unsloth` library, which provides optimized GPU kernels for 4-bit quantized training (~2× faster than standard configurations).

---

## Repository

Training and evaluation code: [a-kishore-dev/llama-text2sql-finetune](https://github.com/a-kishore-dev)

Notebooks included:
- `Text_to_SQL_Finetuning.ipynb` — dataset prep, LoRA config, training, export
- `evaluate_model.ipynb` — ROUGE evaluation comparing base vs fine-tuned

---

## Limitations

- **SQL executability:** ROUGE is a lexical proxy. High ROUGE does not guarantee a query will execute or return logically correct results. A query with different aliases or reordered joins may score lower despite being equivalent.
- **Out-of-distribution schemas:** Performance degrades on high-cardinality databases, deeply nested subqueries, or DDL patterns that diverge significantly from the training distribution.
- **Single epoch:** The model was trained for one epoch on 50k samples. Further training may improve generalization.

---

## License

The model adapter is released under **Apache 2.0**. The underlying base model is governed by the [Meta Llama 3 Community License Agreement](https://llama.meta.com/llama3/license/). Users must comply with both.

---

## Acknowledgements

- **Unsloth** — optimized kernels for 4-bit training and sequence packing
- **Hugging Face** — `trl` (`SFTTrainer`) and `transformers`
- **Meta AI** — Llama 3.2 open weights
- **Gretel AI** — synthetic Text-to-SQL dataset

---

## Author

**A-Kishore** · [GitHub](https://github.com/a-kishore-dev) · [HuggingFace](https://huggingface.co/A-Kishore)