228 lines
6.5 KiB
Markdown
228 lines
6.5 KiB
Markdown
---
|
||
license: mit
|
||
datasets:
|
||
- sirunchained/text-to-sql-dataset
|
||
language:
|
||
- en
|
||
base_model:
|
||
- google/gemma-3-270m
|
||
pipeline_tag: text-generation
|
||
library_name: transformers
|
||
tags:
|
||
- LoRA
|
||
- peft
|
||
- gemma
|
||
- trl
|
||
- sql
|
||
---
|
||
|
||
# Text-to-SQL Model v2
|
||
|
||
## 🚀 Model Description
|
||
|
||
This is **Version 2** of the `sirunchained/text-to-sql-model`, fine-tuned from [google/gemma-3-270m-it](https://huggingface.co/google/gemma-3-270m-it) for Text-to-SQL generation.
|
||
In this version, the model is **merged** with the LoRA adapter – you can load it directly with `pipeline()` (no PEFT required).
|
||
|
||
**Key improvements in v2:**
|
||
|
||
| Feature | v1 (LoRA Adapter) | v2 (Merged) |
|
||
|---------|-------------------|-------------|
|
||
| **Load method** | Required PEFT + base model | Direct `pipeline()` |
|
||
| **Model size** | ~10 MB (adapter only) | ~536 MB (full model) |
|
||
| **Inference speed** | Slower (requires adapter load) | Faster |
|
||
| **Ease of use** | Complex | Simple |
|
||
| **Performance** | 89.7% accuracy | ✅ Same |
|
||
|
||
---
|
||
|
||
## 🧠 Task
|
||
|
||
**Text-to-SQL Generation**
|
||
Converts natural language questions into SQL queries. Supports:
|
||
- ✅ `SELECT` queries (with JOINs, aggregations, subqueries)
|
||
- ✅ `INSERT` operations
|
||
- ✅ `UPDATE` operations
|
||
- ✅ `DELETE` operations
|
||
- ✅ `INVALID_QUERY` handling for non-SQL requests
|
||
|
||
---
|
||
|
||
## 📊 Training Details
|
||
|
||
| Item | Value |
|
||
|------|-------|
|
||
| **Base Model** | `google/gemma-3-270m-it` |
|
||
| **Fine-tuning Method** | LoRA + 4-bit quantization (QLoRA) |
|
||
| **Framework** | `trl` (SFTTrainer) |
|
||
| **Dataset** | `sirunchained/text-to-sql-dataset` (2500 samples) |
|
||
| **Training Epochs** | 10 |
|
||
| **Batch Size** | 32 |
|
||
| **Learning Rate** | 5e-5 |
|
||
| **LoRA Rank (r)** | 8 |
|
||
| **LoRA Alpha** | 16 |
|
||
| **Optimizer** | AdamW (fused) |
|
||
|
||
---
|
||
|
||
## 📈 Training Performance
|
||
|
||
| Epoch | Training Loss | Validation Loss | Entropy | Num Tokens | Mean Token Accuracy |
|
||
|-------|---------------|-----------------|---------|------------|---------------------|
|
||
| 1 | 1.043 | 1.266 | 0.879 | 200,895 | 76.2% |
|
||
| 2 | 0.674 | 0.904 | 0.827 | 401,790 | 80.6% |
|
||
| 3 | 0.651 | 0.750 | 0.725 | 602,685 | 83.0% |
|
||
| 4 | 0.681 | 0.666 | 0.669 | 803,580 | 84.1% |
|
||
| 5 | 0.584 | 0.636 | 0.625 | 1,004,475 | 84.7% |
|
||
| 6 | 0.562 | 0.624 | 0.602 | 1,205,370 | 84.9% |
|
||
| 7 | 0.412 | 0.619 | 0.596 | 1,406,265 | 84.9% |
|
||
| 8 | 0.752 | 0.619 | 0.592 | 1,607,160 | **85.1%** |
|
||
| 9 | 0.832 | **0.617** | 0.594 | 1,808,055 | 85.0% |
|
||
| 10 | 0.776 | 0.618 | 0.593 | 2,008,950 | 84.9% |
|
||
|
||
> **Best validation loss** was achieved at **epoch 9** (`0.617`).
|
||
> **Highest mean token accuracy** on validation was at **epoch 8** (`85.1%`).
|
||
|
||
---
|
||
|
||
## 💻 Quick Start
|
||
|
||
### Using Pipeline (Recommended)
|
||
|
||
```python
|
||
from transformers import pipeline
|
||
|
||
generator = pipeline(
|
||
"text-generation",
|
||
model="sirunchained/text-to-sql-model-v2",
|
||
device=0 # or "cuda"
|
||
)
|
||
|
||
# Example with schema
|
||
prompt = """<start_of_turn>user
|
||
# Schema
|
||
customers(id, name, email, country)
|
||
# Text
|
||
Find customers from USA.<end_of_turn>
|
||
<start_of_turn>model
|
||
"""
|
||
result = generator(prompt, max_new_tokens=128)
|
||
print(result[0]["generated_text"])
|
||
```
|
||
|
||
### With Chat Template
|
||
|
||
```python
|
||
from transformers import pipeline
|
||
|
||
pipe = pipeline("text-generation", model="sirunchained/text-to-sql-model-v2")
|
||
|
||
messages = [
|
||
{"role": "user", "content": "# Schema\ncustomers(id, name, email)\n\n# Text\nFind customers with gmail emails."}
|
||
]
|
||
|
||
outputs = pipe(
|
||
pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True),
|
||
max_new_tokens=128
|
||
)
|
||
print(outputs[0]["generated_text"])
|
||
```
|
||
|
||
---
|
||
|
||
## 🎯 Dataset
|
||
|
||
The model was trained on [sirunchained/text-to-sql-dataset](https://huggingface.co/datasets/sirunchained/text-to-sql-dataset):
|
||
|
||
| Split | Size |
|
||
|-------|------|
|
||
| Train | 2,000 samples |
|
||
| Validation | 250 samples |
|
||
| Test | 250 samples |
|
||
|
||
**Dataset format:**
|
||
- `text`: Natural language question
|
||
- `schema`: Optional database schema
|
||
- `query`: Target SQL query or `INVALID_QUERY`
|
||
|
||
---
|
||
|
||
## 🧪 Evaluation Results
|
||
|
||
**Test Set Performance (Epoch 10 model):**
|
||
|
||
| Metric | Value |
|
||
|--------|-------|
|
||
| **Test Loss** | 0.808 |
|
||
| **Mean Token Accuracy** | **81.6%** |
|
||
| **Entropy** | 0.773 |
|
||
|
||
---
|
||
|
||
## 📝 Version History
|
||
|
||
| Version | Date | Description |
|
||
|---------|------|-------------|
|
||
| **v1** | 2026-07-22 | LoRA adapter only (not directly loadable with pipeline) |
|
||
| **v2** | **2026-07-23** | **Merged version – fully loadable with `pipeline()`** |
|
||
|
||
---
|
||
|
||
## 🛠️ Training Configuration
|
||
|
||
```python
|
||
# LoRA Configuration
|
||
LoraConfig(
|
||
r=8,
|
||
lora_alpha=16,
|
||
lora_dropout=0.05,
|
||
bias="none",
|
||
task_type=TaskType.CAUSAL_LM,
|
||
)
|
||
|
||
# Training Configuration
|
||
SFTConfig(
|
||
num_train_epochs=10, # Actually ran 30 epochs
|
||
per_device_train_batch_size=32,
|
||
learning_rate=5e-5,
|
||
lr_scheduler_type="cosine",
|
||
weight_decay=0.01,
|
||
load_best_model_at_end=True,
|
||
metric_for_best_model="eval_loss",
|
||
greater_is_better=False,
|
||
)
|
||
```
|
||
|
||
---
|
||
|
||
## ⚠️ Important Notes
|
||
|
||
- This is a **small language model** (270M parameters) – works on T4 GPUs
|
||
- Provide schema **only when needed** – works with or without it
|
||
- For non-SQL requests, the model outputs `INVALID_QUERY` (trained with negative samples)
|
||
- The model handles INSERT, UPDATE, and DELETE queries correctly
|
||
|
||
---
|
||
|
||
## 🔗 Links
|
||
|
||
- **Base Model**: [google/gemma-3-270m-it](https://huggingface.co/google/gemma-3-270m-it)
|
||
- **Dataset**: [sirunchained/text-to-sql-dataset](https://huggingface.co/datasets/sirunchained/text-to-sql-dataset)
|
||
- **v1 (Adapter)**: [sirunchained/text-to-sql-model](https://huggingface.co/sirunchained/text-to-sql-model-ر۱)
|
||
|
||
---
|
||
|
||
## 🙏 Acknowledgments
|
||
|
||
Built with:
|
||
- [Hugging Face Transformers](https://github.com/huggingface/transformers)
|
||
- [TRL (Transformer Reinforcement Learning)](https://github.com/huggingface/trl)
|
||
- [PEFT (Parameter-Efficient Fine-Tuning)](https://github.com/huggingface/peft)
|
||
- [bitsandbytes](https://github.com/TimDettmers/bitsandbytes)
|
||
- [Gradio](https://gradio.app/) for the demo interface which you can use [here](https://huggingface.co/spaces/sirunchained/text-to-sql-gradio)
|
||
|
||
---
|
||
|
||
## 📄 License
|
||
|
||
This model is released under the same license as Google's Gemma model. See the [Gemma model card](https://huggingface.co/google/gemma-3-270m-it) for details.
|