68 lines
2.4 KiB
Markdown
68 lines
2.4 KiB
Markdown
|
---
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
base_model: mistralai/Mistral-Nemo-Instruct-2407
|
||
|
|
tags:
|
||
|
|
- text-to-sql
|
||
|
|
- mistral-nemo
|
||
|
|
- spider
|
||
|
|
- peft
|
||
|
|
- qlora
|
||
|
|
metrics:
|
||
|
|
- execution_accuracy
|
||
|
|
- exact_match
|
||
|
|
model_creator: NBAmine
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
datasets:
|
||
|
|
- gretelai/synthetic_text_to_sql
|
||
|
|
- xlangai/spider
|
||
|
|
- NBAmine/xlangai-spider-with-context
|
||
|
|
library_name: transformers
|
||
|
|
---
|
||
|
|
|
||
|
|
# Mistral-Nemo-12B-Text-to-SQL
|
||
|
|
|
||
|
|
[](https://github.com/NBAmine/Nemo-text-to-sql)
|
||
|
|
|
||
|
|
|
||
|
|
## Model Overview
|
||
|
|
This is the full-precision (BF16), merged version of a **Mistral-Nemo-12B** model Parameter-Efficient Fine-Tuned for high-performance **Text-to-SQL** generation. This model is the result of merging LoRA adapters—trained via a two-phase curriculum learning strategy—back into the base weights.
|
||
|
|
|
||
|
|
It is designed to serve as the "Source of Truth" for further optimizations (like AWQ or GGUF) and represents the peak predictive performance of the training pipeline before any quantization-related drift.
|
||
|
|
|
||
|
|
- **Base Model:** `mistralai/Mistral-Nemo-Base-2407`
|
||
|
|
- **Primary Task:** Natural Language to SQL generation with DDL context.
|
||
|
|
- **Output Format:** Standalone SQL queries compatible with standard SQL engines.
|
||
|
|
|
||
|
|
## Training Methodology
|
||
|
|
The model was developed using an MLOps pipeline on dual T4 GPUs in Kaggle.
|
||
|
|
|
||
|
|
### 1. Curriculum Learning Strategy
|
||
|
|
The model underwent a two-stage training process:
|
||
|
|
- **Phase 1 (Syntactic Alignment):** Focused on SQL syntax, basic keywords, and simple schema mapping.
|
||
|
|
- **Phase 2 (Logical Alignment):** Introduced complex reasoning tasks including multiple `JOIN` operations, nested subqueries, and set operations (`UNION`, `INTERSECT`).
|
||
|
|
|
||
|
|
### 2. Fine-Tuning Details
|
||
|
|
- **Technique:** QLoRA (Rank 16, Alpha 32)
|
||
|
|
- **Quantization (during training):** 4-bit NF4
|
||
|
|
- **Optimizer:** Paged AdamW 8-bit
|
||
|
|
- **Hardware:** 2x NVIDIA T4 (Kaggle).
|
||
|
|
|
||
|
|
## Evaluation Results
|
||
|
|
Evaluated on the **Spider** validation set:
|
||
|
|
- **Execution Accuracy (EX):** **69.5%**
|
||
|
|
- **Exact Match (EM):** 61.2%
|
||
|
|
- **Max Context Length:** 2048 tokens
|
||
|
|
|
||
|
|
## Architecture Specs
|
||
|
|
The merged weights utilize the standard Mistral-Nemo 12B architecture:
|
||
|
|
- **Parameters:** 12.2B
|
||
|
|
- **Layers:** 40
|
||
|
|
- **Attention:** Grouped Query Attention (GQA) with 8 KV heads.
|
||
|
|
- **Vocabulary Size:** 128k (Tekken Tokenizer)
|
||
|
|
- **VRAM Requirements:** ~24GB for inference in BF16/FP16.
|
||
|
|
|
||
|
|
|
||
|
|
## Template used during training
|
||
|
|
|
||
|
|
prompt = "Context: {DDL}<br>Question: {NL_QUERY}<br>Answer:"
|