初始化项目,由ModelHub XC社区提供模型
Model: NBAmine/mistral-nemo-text-to-sql Source: Original Platform
This commit is contained in:
68
README.md
Normal file
68
README.md
Normal file
@@ -0,0 +1,68 @@
|
||||
---
|
||||
language:
|
||||
- en
|
||||
base_model: mistralai/Mistral-Nemo-Instruct-2407
|
||||
tags:
|
||||
- text-to-sql
|
||||
- mistral-nemo
|
||||
- spider
|
||||
- peft
|
||||
- qlora
|
||||
metrics:
|
||||
- execution_accuracy
|
||||
- exact_match
|
||||
model_creator: NBAmine
|
||||
pipeline_tag: text-generation
|
||||
datasets:
|
||||
- gretelai/synthetic_text_to_sql
|
||||
- xlangai/spider
|
||||
- NBAmine/xlangai-spider-with-context
|
||||
library_name: transformers
|
||||
---
|
||||
|
||||
# Mistral-Nemo-12B-Text-to-SQL
|
||||
|
||||
[](https://github.com/NBAmine/Nemo-text-to-sql)
|
||||
|
||||
|
||||
## Model Overview
|
||||
This is the full-precision (BF16), merged version of a **Mistral-Nemo-12B** model Parameter-Efficient Fine-Tuned for high-performance **Text-to-SQL** generation. This model is the result of merging LoRA adapters—trained via a two-phase curriculum learning strategy—back into the base weights.
|
||||
|
||||
It is designed to serve as the "Source of Truth" for further optimizations (like AWQ or GGUF) and represents the peak predictive performance of the training pipeline before any quantization-related drift.
|
||||
|
||||
- **Base Model:** `mistralai/Mistral-Nemo-Base-2407`
|
||||
- **Primary Task:** Natural Language to SQL generation with DDL context.
|
||||
- **Output Format:** Standalone SQL queries compatible with standard SQL engines.
|
||||
|
||||
## Training Methodology
|
||||
The model was developed using an MLOps pipeline on dual T4 GPUs in Kaggle.
|
||||
|
||||
### 1. Curriculum Learning Strategy
|
||||
The model underwent a two-stage training process:
|
||||
- **Phase 1 (Syntactic Alignment):** Focused on SQL syntax, basic keywords, and simple schema mapping.
|
||||
- **Phase 2 (Logical Alignment):** Introduced complex reasoning tasks including multiple `JOIN` operations, nested subqueries, and set operations (`UNION`, `INTERSECT`).
|
||||
|
||||
### 2. Fine-Tuning Details
|
||||
- **Technique:** QLoRA (Rank 16, Alpha 32)
|
||||
- **Quantization (during training):** 4-bit NF4
|
||||
- **Optimizer:** Paged AdamW 8-bit
|
||||
- **Hardware:** 2x NVIDIA T4 (Kaggle).
|
||||
|
||||
## Evaluation Results
|
||||
Evaluated on the **Spider** validation set:
|
||||
- **Execution Accuracy (EX):** **69.5%**
|
||||
- **Exact Match (EM):** 61.2%
|
||||
- **Max Context Length:** 2048 tokens
|
||||
|
||||
## Architecture Specs
|
||||
The merged weights utilize the standard Mistral-Nemo 12B architecture:
|
||||
- **Parameters:** 12.2B
|
||||
- **Layers:** 40
|
||||
- **Attention:** Grouped Query Attention (GQA) with 8 KV heads.
|
||||
- **Vocabulary Size:** 128k (Tekken Tokenizer)
|
||||
- **VRAM Requirements:** ~24GB for inference in BF16/FP16.
|
||||
|
||||
|
||||
## Template used during training
|
||||
|
||||
prompt = "Context: {DDL}<br>Question: {NL_QUERY}<br>Answer:"
|
||||
Reference in New Issue
Block a user