---
license: mit
datasets:
- sirunchained/text-to-sql-dataset
language:
- en
base_model:
- google/gemma-3-270m
pipeline_tag: text-generation
library_name: transformers
tags:
- LoRA
- peft
- gemma
- trl
- sql
---

# Text-to-SQL Model v2

## 🚀 Model Description

This is **Version 2** of the `sirunchained/text-to-sql-model`, fine-tuned from [google/gemma-3-270m-it](https://huggingface.co/google/gemma-3-270m-it) for Text-to-SQL generation.  
In this version, the model is **merged** with the LoRA adapter – you can load it directly with `pipeline()` (no PEFT required).

**Key improvements in v2:**

| Feature | v1 (LoRA Adapter) | v2 (Merged) |
|---------|-------------------|-------------|
| **Load method** | Required PEFT + base model | Direct `pipeline()` |
| **Model size** | ~10 MB (adapter only) | ~536 MB (full model) |
| **Inference speed** | Slower (requires adapter load) | Faster |
| **Ease of use** | Complex | Simple |
| **Performance** | 89.7% accuracy | ✅ Same |

---

## 🧠 Task

**Text-to-SQL Generation**  
Converts natural language questions into SQL queries. Supports:
- ✅ `SELECT` queries (with JOINs, aggregations, subqueries)
- ✅ `INSERT` operations
- ✅ `UPDATE` operations
- ✅ `DELETE` operations
- ✅ `INVALID_QUERY` handling for non-SQL requests

---

## 📊 Training Details

| Item | Value |
|------|-------|
| **Base Model** | `google/gemma-3-270m-it` |
| **Fine-tuning Method** | LoRA + 4-bit quantization (QLoRA) |
| **Framework** | `trl` (SFTTrainer) |
| **Dataset** | `sirunchained/text-to-sql-dataset` (2500 samples) |
| **Training Epochs** | 10 |
| **Batch Size** | 32 |
| **Learning Rate** | 5e-5 |
| **LoRA Rank (r)** | 8 |
| **LoRA Alpha** | 16 |
| **Optimizer** | AdamW (fused) |

---

## 📈 Training Performance

| Epoch | Training Loss | Validation Loss | Entropy | Num Tokens | Mean Token Accuracy |
|-------|---------------|-----------------|---------|------------|---------------------|
| 1     | 1.043         | 1.266           | 0.879   | 200,895    | 76.2%               |
| 2     | 0.674         | 0.904           | 0.827   | 401,790    | 80.6%               |
| 3     | 0.651         | 0.750           | 0.725   | 602,685    | 83.0%               |
| 4     | 0.681         | 0.666           | 0.669   | 803,580    | 84.1%               |
| 5     | 0.584         | 0.636           | 0.625   | 1,004,475  | 84.7%               |
| 6     | 0.562         | 0.624           | 0.602   | 1,205,370  | 84.9%               |
| 7     | 0.412         | 0.619           | 0.596   | 1,406,265  | 84.9%               |
| 8     | 0.752         | 0.619           | 0.592   | 1,607,160  | **85.1%**           |
| 9     | 0.832         | **0.617**       | 0.594   | 1,808,055  | 85.0%               |
| 10    | 0.776         | 0.618           | 0.593   | 2,008,950  | 84.9%               |

> **Best validation loss** was achieved at **epoch 9** (`0.617`).  
> **Highest mean token accuracy** on validation was at **epoch 8** (`85.1%`).

---

## 💻 Quick Start

### Using Pipeline (Recommended)

```python
from transformers import pipeline

generator = pipeline(
    "text-generation",
    model="sirunchained/text-to-sql-model-v2",
    device=0  # or "cuda"
)

# Example with schema
prompt = """<start_of_turn>user
# Schema
customers(id, name, email, country)
# Text
Find customers from USA.<end_of_turn>
<start_of_turn>model
"""
result = generator(prompt, max_new_tokens=128)
print(result[0]["generated_text"])
```

### With Chat Template

```python
from transformers import pipeline

pipe = pipeline("text-generation", model="sirunchained/text-to-sql-model-v2")

messages = [
    {"role": "user", "content": "# Schema\ncustomers(id, name, email)\n\n# Text\nFind customers with gmail emails."}
]

outputs = pipe(
    pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True),
    max_new_tokens=128
)
print(outputs[0]["generated_text"])
```

---

## 🎯 Dataset

The model was trained on [sirunchained/text-to-sql-dataset](https://huggingface.co/datasets/sirunchained/text-to-sql-dataset):

| Split | Size |
|-------|------|
| Train | 2,000 samples |
| Validation | 250 samples |
| Test | 250 samples |

**Dataset format:**
- `text`: Natural language question
- `schema`: Optional database schema
- `query`: Target SQL query or `INVALID_QUERY`

---

## 🧪 Evaluation Results

**Test Set Performance (Epoch 10 model):**

| Metric | Value |
|--------|-------|
| **Test Loss** | 0.808 |
| **Mean Token Accuracy** | **81.6%** |
| **Entropy** | 0.773 |

---

## 📝 Version History

| Version | Date | Description |
|---------|------|-------------|
| **v1** | 2026-07-22 | LoRA adapter only (not directly loadable with pipeline) |
| **v2** | **2026-07-23** | **Merged version – fully loadable with `pipeline()`** |

---

## 🛠️ Training Configuration

```python
# LoRA Configuration
LoraConfig(
    r=8,
    lora_alpha=16,
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM,
)

# Training Configuration
SFTConfig(
    num_train_epochs=10,  # Actually ran 30 epochs
    per_device_train_batch_size=32,
    learning_rate=5e-5,
    lr_scheduler_type="cosine",
    weight_decay=0.01,
    load_best_model_at_end=True,
    metric_for_best_model="eval_loss",
    greater_is_better=False,
)
```

---

## ⚠️ Important Notes

- This is a **small language model** (270M parameters) – works on T4 GPUs
- Provide schema **only when needed** – works with or without it
- For non-SQL requests, the model outputs `INVALID_QUERY` (trained with negative samples)
- The model handles INSERT, UPDATE, and DELETE queries correctly

---

## 🔗 Links

- **Base Model**: [google/gemma-3-270m-it](https://huggingface.co/google/gemma-3-270m-it)
- **Dataset**: [sirunchained/text-to-sql-dataset](https://huggingface.co/datasets/sirunchained/text-to-sql-dataset)
- **v1 (Adapter)**: [sirunchained/text-to-sql-model](https://huggingface.co/sirunchained/text-to-sql-model-ر۱)

---

## 🙏 Acknowledgments

Built with:
- [Hugging Face Transformers](https://github.com/huggingface/transformers)
- [TRL (Transformer Reinforcement Learning)](https://github.com/huggingface/trl)
- [PEFT (Parameter-Efficient Fine-Tuning)](https://github.com/huggingface/peft)
- [bitsandbytes](https://github.com/TimDettmers/bitsandbytes)
- [Gradio](https://gradio.app/) for the demo interface which you can use [here](https://huggingface.co/spaces/sirunchained/text-to-sql-gradio)

---

## 📄 License

This model is released under the same license as Google's Gemma model. See the [Gemma model card](https://huggingface.co/google/gemma-3-270m-it) for details.