初始化项目,由ModelHub XC社区提供模型
Model: ashishsahu2008/qwen2.5-3b-text2sql Source: Original Platform
This commit is contained in:
113
README.md
Normal file
113
README.md
Normal file
@@ -0,0 +1,113 @@
|
||||
---
|
||||
base_model: unsloth/Qwen2.5-3B-Instruct
|
||||
library_name: transformers
|
||||
pipeline_tag: text-generation
|
||||
license: apache-2.0
|
||||
language:
|
||||
- en
|
||||
tags:
|
||||
- text-to-sql
|
||||
- sql
|
||||
- qwen2
|
||||
- lora
|
||||
- qlora
|
||||
- unsloth
|
||||
- trl
|
||||
datasets:
|
||||
- b-mc2/sql-create-context
|
||||
---
|
||||
|
||||
# Qwen2.5-3B Text-to-SQL
|
||||
|
||||
A fine-tuned version of **Qwen2.5-3B-Instruct** that converts natural-language
|
||||
questions into SQL queries, given a database schema. Trained with LoRA/QLoRA on a
|
||||
single free GPU.
|
||||
|
||||
- **Developed by:** ashishsahu2008
|
||||
- **Base model:** [unsloth/Qwen2.5-3B-Instruct](https://huggingface.co/unsloth/Qwen2.5-3B-Instruct)
|
||||
- **Fine-tuned on:** [b-mc2/sql-create-context](https://huggingface.co/datasets/b-mc2/sql-create-context)
|
||||
- **License:** apache-2.0
|
||||
- **Live demo:** https://huggingface.co/spaces/ashishsahu2008/text2sql-demo
|
||||
- **Code / write-up:** https://github.com/ashishsahu2008/fine-tuning
|
||||
|
||||
## What it does
|
||||
|
||||
Give it a `CREATE TABLE` schema and a plain-English question, and it returns only
|
||||
the SQL query that answers the question — no explanations or markdown, just runnable SQL.
|
||||
|
||||
## Results
|
||||
|
||||
Evaluated on a held-out test split (n=200) the model never saw during training.
|
||||
Both the base and fine-tuned models were scored with identical string normalization
|
||||
(lowercase, strip `;`, canonicalize quotes), so the comparison is apples-to-apples.
|
||||
|
||||
| Model | Exact-match accuracy |
|
||||
|--------------------------------|:--------------------:|
|
||||
| Qwen2.5-3B-Instruct (base) | 41.5% |
|
||||
| **This model (fine-tuned)** | **72.5%** |
|
||||
|
||||
Only 0.96% of parameters (~30M of 3.1B) were trained via LoRA. Much of the gain
|
||||
comes from the base model learning to emit clean, executable SQL instead of wrapping
|
||||
answers in prose or markdown fences.
|
||||
|
||||
> **Note on the metric:** exact-match undercounts semantically-correct queries that
|
||||
> are written differently (column order, aliases, whitespace). A stronger evaluation
|
||||
> would use AST comparison (`sqlglot`) or execution accuracy on a benchmark with
|
||||
> populated databases (Spider / BIRD); `sql-create-context` ships schemas only.
|
||||
|
||||
## Usage
|
||||
|
||||
```python
|
||||
import torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
model_id = "ashishsahu2008/qwen2.5-3b-text2sql"
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16)
|
||||
|
||||
schema = "CREATE TABLE employees (name VARCHAR, salary INTEGER, department VARCHAR)"
|
||||
question = "List the names of employees in Sales earning over 50000."
|
||||
|
||||
messages = [
|
||||
{"role": "system", "content": "You are a SQL expert. Given a database schema "
|
||||
"and a question, output only the SQL query that answers it."},
|
||||
{"role": "user", "content": f"Schema:\n{schema}\n\nQuestion: {question}"},
|
||||
]
|
||||
inputs = tokenizer.apply_chat_template(
|
||||
messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
|
||||
).to(model.device)
|
||||
out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
|
||||
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
|
||||
```
|
||||
|
||||
## Prompt format
|
||||
|
||||
The model expects the schema and question in a single user turn, with this system prompt:
|
||||
|
||||
```
|
||||
System: You are a SQL expert. Given a database schema and a question, output only the SQL query that answers it.
|
||||
User: Schema:
|
||||
<CREATE TABLE ...>
|
||||
|
||||
Question: <your question>
|
||||
```
|
||||
|
||||
## Training details
|
||||
|
||||
- **Method:** supervised fine-tuning (SFT) with LoRA on a 4-bit quantized base (QLoRA)
|
||||
- **LoRA config:** rank 16, alpha 16, applied to attention + MLP projections
|
||||
- **Data:** 3,000-row subset of `b-mc2/sql-create-context` (2,700 train / 300 test)
|
||||
- **Hyperparameters:** 2 epochs, learning rate 2e-4, effective batch size 8
|
||||
- **Hardware:** single Colab T4, ~25 minutes
|
||||
- **Frameworks:** [Unsloth](https://github.com/unslothai/unsloth) + TRL `SFTTrainer`
|
||||
|
||||
## Limitations
|
||||
|
||||
- Trained on **single-table** `CREATE TABLE` schemas; complex multi-join databases
|
||||
are out of distribution.
|
||||
- Assumes the schema is provided in the prompt — it does not know any tables you
|
||||
don't give it.
|
||||
- SQL dialect follows the training data (broadly SQLite-compatible); it does not
|
||||
target a specific engine's extensions.
|
||||
|
||||
Fine-tuned with [Unsloth](https://github.com/unslothai/unsloth) and Hugging Face's TRL library.
|
||||
Reference in New Issue
Block a user