初始化项目，由ModelHub XC社区提供模型
Model: A-Kishore/llama-3.2-3b-text2sql Source: Original Platform
This commit is contained in:
ModelHub XC
253
README.md
Normal file
253
README.md
Normal file
@@ -0,0 +1,253 @@
|
||||
---
|
||||
base_model: unsloth/Llama-3.2-3B-Instruct-bnb-4bit
|
||||
tags:
|
||||
- text-generation-inference
|
||||
- transformers
|
||||
- unsloth
|
||||
- llama
|
||||
- text-to-sql
|
||||
- peft
|
||||
- lora
|
||||
license: apache-2.0
|
||||
language:
|
||||
- en
|
||||
datasets:
|
||||
- gretelai/synthetic_text_to_sql
|
||||
---
|
||||
|
||||
# 🔮 Llama-3.2-3B-Instruct Text-to-SQL
|
||||
|
||||
A fine-tuned version of [`unsloth/Llama-3.2-3B-Instruct-bnb-4bit`](https://huggingface.co/unsloth/Llama-3.2-3B-Instruct-bnb-4bit) for generating SQL queries from natural language questions and database DDL schemas.
|
||||
|
||||
[](https://opensource.org/licenses/Apache-2.0)
|
||||
[](https://github.com/unslothai/unsloth)
|
||||
|
||||
---
|
||||
|
||||
## Model Summary
|
||||
|
||||
| Attribute | Value |
|
||||
|:---|:---|
|
||||
| **Base Model** | `unsloth/Llama-3.2-3B-Instruct-bnb-4bit` |
|
||||
| **Task** | Text-to-SQL (natural language → SQL) |
|
||||
| **Fine-Tuning Method** | LoRA (PEFT) via Unsloth + TRL |
|
||||
| **Training Dataset** | `gretelai/synthetic_text_to_sql` (50k samples) |
|
||||
| **Trainable Parameters** | ~0.75% of base model |
|
||||
| **Export Format** | Merged FP16 (`merged_16bit`) |
|
||||
| **License** | Apache-2.0 + Meta Llama 3 Community License |
|
||||
| **Developer** | A-Kishore |
|
||||
|
||||
---
|
||||
|
||||
## Evaluation Results
|
||||
|
||||
Evaluated on the first 200 samples of the `gretelai/synthetic_text_to_sql` test split using greedy decoding. ROUGE F-measures reported.
|
||||
|
||||
| Model | ROUGE-1 | ROUGE-2 | ROUGE-L |
|
||||
|:---|:---:|:---:|:---:|
|
||||
| Base Model (`unsloth/Llama-3.2-3B-Instruct-bnb-4bit`) | 0.2908 | 0.2016 | 0.2651 |
|
||||
| Fine-Tuned (`A-Kishore/llama-3.2-3b-text2sql`) | **0.8486** | **0.7232** | **0.8151** |
|
||||
| **Improvement** | **+191.82%** | **+258.73%** | **+207.47%** |
|
||||
|
||||
**Metric interpretation:**
|
||||
- **ROUGE-1** (unigram overlap) reflects accurate retrieval of schema identifiers and SQL keywords.
|
||||
- **ROUGE-2** (bigram overlap) captures structural alignment of consecutive SQL constructs (e.g. `GROUP BY`, `ORDER BY`).
|
||||
- **ROUGE-L** (longest common subsequence) tracks overall query flow including nested clauses and join ordering.
|
||||
|
||||
> **Note:** ROUGE measures lexical overlap, not SQL executability. A query may score slightly lower due to stylistic differences (alias names, join ordering) while still being functionally equivalent. See [Limitations](#limitations).
|
||||
|
||||
---
|
||||
|
||||
## How to Use
|
||||
|
||||
The model weights are fully merged in 16-bit precision and load with standard `transformers` or `unsloth`.
|
||||
|
||||
### Prompt Format
|
||||
|
||||
Always use this exact template — the model was trained on it:
|
||||
|
||||
```
|
||||
###TASK
|
||||
Generate the SQL query to answer the following question
|
||||
|
||||
### Database Schema
|
||||
{sql_context}
|
||||
|
||||
### Question
|
||||
{sql_prompt}
|
||||
|
||||
### SQL Query
|
||||
```
|
||||
|
||||
### (a) Standard `transformers`
|
||||
|
||||
```python
|
||||
import torch
|
||||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||||
|
||||
model_name = "A-Kishore/llama-3.2-3b-text2sql"
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
model_name,
|
||||
torch_dtype=torch.float16,
|
||||
device_map="auto"
|
||||
)
|
||||
|
||||
prompt = """###TASK
|
||||
Generate the SQL query to answer the following question
|
||||
|
||||
### Database Schema
|
||||
{sql_context}
|
||||
|
||||
### Question
|
||||
{sql_prompt}
|
||||
|
||||
### SQL Query
|
||||
"""
|
||||
|
||||
sql_context = "CREATE TABLE employees (id INT, name TEXT, department TEXT, salary REAL);"
|
||||
sql_prompt = "What is the average salary per department?"
|
||||
|
||||
inputs = tokenizer(
|
||||
prompt.format(sql_context=sql_context, sql_prompt=sql_prompt),
|
||||
return_tensors="pt"
|
||||
).to("cuda")
|
||||
|
||||
outputs = model.generate(
|
||||
**inputs,
|
||||
max_new_tokens=150,
|
||||
use_cache=True,
|
||||
pad_token_id=tokenizer.eos_token_id
|
||||
)
|
||||
|
||||
result = tokenizer.decode(outputs[0], skip_special_tokens=True)
|
||||
sql = result.split("### SQL Query")[-1].strip()
|
||||
print(sql)
|
||||
# SELECT department, AVG(salary) FROM employees GROUP BY department;
|
||||
```
|
||||
|
||||
### (b) Unsloth Fast Inference
|
||||
|
||||
```python
|
||||
import torch
|
||||
from unsloth import FastLanguageModel
|
||||
|
||||
model, tokenizer = FastLanguageModel.from_pretrained(
|
||||
model_name="A-Kishore/llama-3.2-3b-text2sql",
|
||||
max_seq_length=768,
|
||||
dtype=torch.float16,
|
||||
load_in_4bit=False,
|
||||
)
|
||||
FastLanguageModel.for_inference(model)
|
||||
|
||||
prompt = """###TASK
|
||||
Generate the SQL query to answer the following question
|
||||
|
||||
### Database Schema
|
||||
{sql_context}
|
||||
|
||||
### Question
|
||||
{sql_prompt}
|
||||
|
||||
### SQL Query
|
||||
"""
|
||||
|
||||
sql_context = "CREATE TABLE employees (id INT, name TEXT, department TEXT, salary REAL);"
|
||||
sql_prompt = "What is the average salary per department?"
|
||||
|
||||
inputs = tokenizer(
|
||||
prompt.format(sql_context=sql_context, sql_prompt=sql_prompt),
|
||||
return_tensors="pt"
|
||||
).to("cuda")
|
||||
|
||||
outputs = model.generate(
|
||||
**inputs,
|
||||
max_new_tokens=150,
|
||||
temperature=None,
|
||||
do_sample=False,
|
||||
pad_token_id=tokenizer.eos_token_id
|
||||
)
|
||||
|
||||
result = tokenizer.decode(outputs[0], skip_special_tokens=True)
|
||||
sql = result.split("### SQL Query")[-1].strip()
|
||||
print(sql)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Training Details
|
||||
|
||||
### Dataset
|
||||
|
||||
- **Source:** [`gretelai/synthetic_text_to_sql`](https://huggingface.co/datasets/gretelai/synthetic_text_to_sql)
|
||||
- **Training split:** 50,000 shuffled samples from `train`
|
||||
- **Evaluation split:** First 200 samples from `test`
|
||||
|
||||
### LoRA Configuration
|
||||
|
||||
| Parameter | Value |
|
||||
|:---|:---|
|
||||
| Rank (`r`) | `16` |
|
||||
| Alpha (`lora_alpha`) | `16` |
|
||||
| Target Modules | `q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj` |
|
||||
| Bias | `none` |
|
||||
|
||||
LoRA freezes the base model weights and injects trainable rank-decomposition matrices into all attention and MLP projections. Only ~0.75% of parameters are updated, dramatically reducing VRAM usage and preventing catastrophic forgetting.
|
||||
|
||||
### Hyperparameters
|
||||
|
||||
| Parameter | Value |
|
||||
|:---|:---|
|
||||
| Optimizer | `paged_adamw_8bit` |
|
||||
| Learning Rate | `2e-4` |
|
||||
| LR Scheduler | `linear` |
|
||||
| Warmup Steps | `5` |
|
||||
| Epochs | `1` |
|
||||
| Per-Device Batch Size | `8` |
|
||||
| Gradient Accumulation | `1` |
|
||||
| Max Sequence Length | `768` |
|
||||
| Sequence Packing | `True` |
|
||||
| Mixed Precision | `fp16` |
|
||||
| Experiment Tracking | Weights & Biases |
|
||||
|
||||
Training was accelerated using the `unsloth` library, which provides optimized GPU kernels for 4-bit quantized training (~2× faster than standard configurations).
|
||||
|
||||
---
|
||||
|
||||
## Repository
|
||||
|
||||
Training and evaluation code: [a-kishore-dev/llama-text2sql-finetune](https://github.com/a-kishore-dev)
|
||||
|
||||
Notebooks included:
|
||||
- `Text_to_SQL_Finetuning.ipynb` — dataset prep, LoRA config, training, export
|
||||
- `evaluate_model.ipynb` — ROUGE evaluation comparing base vs fine-tuned
|
||||
|
||||
---
|
||||
|
||||
## Limitations
|
||||
|
||||
- **SQL executability:** ROUGE is a lexical proxy. High ROUGE does not guarantee a query will execute or return logically correct results. A query with different aliases or reordered joins may score lower despite being equivalent.
|
||||
- **Out-of-distribution schemas:** Performance degrades on high-cardinality databases, deeply nested subqueries, or DDL patterns that diverge significantly from the training distribution.
|
||||
- **Single epoch:** The model was trained for one epoch on 50k samples. Further training may improve generalization.
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
The model adapter is released under **Apache 2.0**. The underlying base model is governed by the [Meta Llama 3 Community License Agreement](https://llama.meta.com/llama3/license/). Users must comply with both.
|
||||
|
||||
---
|
||||
|
||||
## Acknowledgements
|
||||
|
||||
- **Unsloth** — optimized kernels for 4-bit training and sequence packing
|
||||
- **Hugging Face** — `trl` (`SFTTrainer`) and `transformers`
|
||||
- **Meta AI** — Llama 3.2 open weights
|
||||
- **Gretel AI** — synthetic Text-to-SQL dataset
|
||||
|
||||
---
|
||||
|
||||
## Author
|
||||
|
||||
**A-Kishore** · [GitHub](https://github.com/a-kishore-dev) · [HuggingFace](https://huggingface.co/A-Kishore)
|
||||
Reference in New Issue
Block a user