commit e2a3d0c21e99726de5576dc24cee8041a5a9e8e7 Author: ModelHub XC Date: Mon Jul 27 22:26:10 2026 +0800 初始化项目，由ModelHub XC社区提供模型 Model: sirunchained/text-to-sql-model-v2 Source: Original Platform diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..8ccc879 --- /dev/null +++ b/.gitattributes @@ -0,0 +1,37 @@ +*.7z filter=lfs diff=lfs merge=lfs -text +*.arrow filter=lfs diff=lfs merge=lfs -text +*.bin filter=lfs diff=lfs merge=lfs -text +*.bz2 filter=lfs diff=lfs merge=lfs -text +*.ckpt filter=lfs diff=lfs merge=lfs -text +*.ftz filter=lfs diff=lfs merge=lfs -text +*.gz filter=lfs diff=lfs merge=lfs -text +*.h5 filter=lfs diff=lfs merge=lfs -text +*.joblib filter=lfs diff=lfs merge=lfs -text +*.lfs.* filter=lfs diff=lfs merge=lfs -text +*.mlmodel filter=lfs diff=lfs merge=lfs -text +*.model filter=lfs diff=lfs merge=lfs -text +*.msgpack filter=lfs diff=lfs merge=lfs -text +*.npy filter=lfs diff=lfs merge=lfs -text +*.npz filter=lfs diff=lfs merge=lfs -text +*.onnx filter=lfs diff=lfs merge=lfs -text +*.ot filter=lfs diff=lfs merge=lfs -text +*.parquet filter=lfs diff=lfs merge=lfs -text +*.pb filter=lfs diff=lfs merge=lfs -text +*.pickle filter=lfs diff=lfs merge=lfs -text +*.pkl filter=lfs diff=lfs merge=lfs -text +*.pt filter=lfs diff=lfs merge=lfs -text +*.pth filter=lfs diff=lfs merge=lfs -text +*.rar filter=lfs diff=lfs merge=lfs -text +*.safetensors filter=lfs diff=lfs merge=lfs -text +saved_model/**/* filter=lfs diff=lfs merge=lfs -text +*.tar.* filter=lfs diff=lfs merge=lfs -text +*.tar filter=lfs diff=lfs merge=lfs -text +*.tflite filter=lfs diff=lfs merge=lfs -text +*.tgz filter=lfs diff=lfs merge=lfs -text +*.wasm filter=lfs diff=lfs merge=lfs -text +*.xz filter=lfs diff=lfs merge=lfs -text +*.zip filter=lfs diff=lfs merge=lfs -text +*.zst filter=lfs diff=lfs merge=lfs -text +*tfevents* filter=lfs diff=lfs merge=lfs -text +lora-adapter/tokenizer.json filter=lfs diff=lfs merge=lfs -text +tokenizer.json filter=lfs diff=lfs merge=lfs -text diff --git a/README.md b/README.md new file mode 100644 index 0000000..741d3eb --- /dev/null +++ b/README.md @@ -0,0 +1,227 @@ +--- +license: mit +datasets: +- sirunchained/text-to-sql-dataset +language: +- en +base_model: +- google/gemma-3-270m +pipeline_tag: text-generation +library_name: transformers +tags: +- LoRA +- peft +- gemma +- trl +- sql +--- + +# Text-to-SQL Model v2 + +## 🚀 Model Description + +This is **Version 2** of the `sirunchained/text-to-sql-model`, fine-tuned from [google/gemma-3-270m-it](https://huggingface.co/google/gemma-3-270m-it) for Text-to-SQL generation. +In this version, the model is **merged** with the LoRA adapter – you can load it directly with `pipeline()` (no PEFT required). + +**Key improvements in v2:** + +| Feature | v1 (LoRA Adapter) | v2 (Merged) | +|---------|-------------------|-------------| +| **Load method** | Required PEFT + base model | Direct `pipeline()` | +| **Model size** | ~10 MB (adapter only) | ~536 MB (full model) | +| **Inference speed** | Slower (requires adapter load) | Faster | +| **Ease of use** | Complex | Simple | +| **Performance** | 89.7% accuracy | ✅ Same | + +--- + +## 🧠 Task + +**Text-to-SQL Generation** +Converts natural language questions into SQL queries. Supports: +- ✅ `SELECT` queries (with JOINs, aggregations, subqueries) +- ✅ `INSERT` operations +- ✅ `UPDATE` operations +- ✅ `DELETE` operations +- ✅ `INVALID_QUERY` handling for non-SQL requests + +--- + +## 📊 Training Details + +| Item | Value | +|------|-------| +| **Base Model** | `google/gemma-3-270m-it` | +| **Fine-tuning Method** | LoRA + 4-bit quantization (QLoRA) | +| **Framework** | `trl` (SFTTrainer) | +| **Dataset** | `sirunchained/text-to-sql-dataset` (2500 samples) | +| **Training Epochs** | 10 | +| **Batch Size** | 32 | +| **Learning Rate** | 5e-5 | +| **LoRA Rank (r)** | 8 | +| **LoRA Alpha** | 16 | +| **Optimizer** | AdamW (fused) | + +--- + +## 📈 Training Performance + +| Epoch | Training Loss | Validation Loss | Entropy | Num Tokens | Mean Token Accuracy | +|-------|---------------|-----------------|---------|------------|---------------------| +| 1 | 1.043 | 1.266 | 0.879 | 200,895 | 76.2% | +| 2 | 0.674 | 0.904 | 0.827 | 401,790 | 80.6% | +| 3 | 0.651 | 0.750 | 0.725 | 602,685 | 83.0% | +| 4 | 0.681 | 0.666 | 0.669 | 803,580 | 84.1% | +| 5 | 0.584 | 0.636 | 0.625 | 1,004,475 | 84.7% | +| 6 | 0.562 | 0.624 | 0.602 | 1,205,370 | 84.9% | +| 7 | 0.412 | 0.619 | 0.596 | 1,406,265 | 84.9% | +| 8 | 0.752 | 0.619 | 0.592 | 1,607,160 | **85.1%** | +| 9 | 0.832 | **0.617** | 0.594 | 1,808,055 | 85.0% | +| 10 | 0.776 | 0.618 | 0.593 | 2,008,950 | 84.9% | + +> **Best validation loss** was achieved at **epoch 9** (`0.617`). +> **Highest mean token accuracy** on validation was at **epoch 8** (`85.1%`). + +--- + +## 💻 Quick Start + +### Using Pipeline (Recommended) + +```python +from transformers import pipeline + +generator = pipeline( + "text-generation", + model="sirunchained/text-to-sql-model-v2", + device=0 # or "cuda" +) + +# Example with schema +prompt = """user +# Schema +customers(id, name, email, country) +# Text +Find customers from USA. +model +""" +result = generator(prompt, max_new_tokens=128) +print(result[0]["generated_text"]) +``` + +### With Chat Template + +```python +from transformers import pipeline + +pipe = pipeline("text-generation", model="sirunchained/text-to-sql-model-v2") + +messages = [ + {"role": "user", "content": "# Schema\ncustomers(id, name, email)\n\n# Text\nFind customers with gmail emails."} +] + +outputs = pipe( + pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True), + max_new_tokens=128 +) +print(outputs[0]["generated_text"]) +``` + +--- + +## 🎯 Dataset + +The model was trained on [sirunchained/text-to-sql-dataset](https://huggingface.co/datasets/sirunchained/text-to-sql-dataset): + +| Split | Size | +|-------|------| +| Train | 2,000 samples | +| Validation | 250 samples | +| Test | 250 samples | + +**Dataset format:** +- `text`: Natural language question +- `schema`: Optional database schema +- `query`: Target SQL query or `INVALID_QUERY` + +--- + +## 🧪 Evaluation Results + +**Test Set Performance (Epoch 10 model):** + +| Metric | Value | +|--------|-------| +| **Test Loss** | 0.808 | +| **Mean Token Accuracy** | **81.6%** | +| **Entropy** | 0.773 | + +--- + +## 📝 Version History + +| Version | Date | Description | +|---------|------|-------------| +| **v1** | 2026-07-22 | LoRA adapter only (not directly loadable with pipeline) | +| **v2** | **2026-07-23** | **Merged version – fully loadable with `pipeline()`** | + +--- + +## 🛠️ Training Configuration + +```python +# LoRA Configuration +LoraConfig( + r=8, + lora_alpha=16, + lora_dropout=0.05, + bias="none", + task_type=TaskType.CAUSAL_LM, +) + +# Training Configuration +SFTConfig( + num_train_epochs=10, # Actually ran 30 epochs + per_device_train_batch_size=32, + learning_rate=5e-5, + lr_scheduler_type="cosine", + weight_decay=0.01, + load_best_model_at_end=True, + metric_for_best_model="eval_loss", + greater_is_better=False, +) +``` + +--- + +## ⚠️ Important Notes + +- This is a **small language model** (270M parameters) – works on T4 GPUs +- Provide schema **only when needed** – works with or without it +- For non-SQL requests, the model outputs `INVALID_QUERY` (trained with negative samples) +- The model handles INSERT, UPDATE, and DELETE queries correctly + +--- + +## 🔗 Links + +- **Base Model**: [google/gemma-3-270m-it](https://huggingface.co/google/gemma-3-270m-it) +- **Dataset**: [sirunchained/text-to-sql-dataset](https://huggingface.co/datasets/sirunchained/text-to-sql-dataset) +- **v1 (Adapter)**: [sirunchained/text-to-sql-model](https://huggingface.co/sirunchained/text-to-sql-model-ر۱) + +--- + +## 🙏 Acknowledgments + +Built with: +- [Hugging Face Transformers](https://github.com/huggingface/transformers) +- [TRL (Transformer Reinforcement Learning)](https://github.com/huggingface/trl) +- [PEFT (Parameter-Efficient Fine-Tuning)](https://github.com/huggingface/peft) +- [bitsandbytes](https://github.com/TimDettmers/bitsandbytes) +- [Gradio](https://gradio.app/) for the demo interface which you can use [here](https://huggingface.co/spaces/sirunchained/text-to-sql-gradio) + +--- + +## 📄 License + +This model is released under the same license as Google's Gemma model. See the [Gemma model card](https://huggingface.co/google/gemma-3-270m-it) for details. diff --git a/chat_template.jinja b/chat_template.jinja new file mode 100644 index 0000000..1117055 --- /dev/null +++ b/chat_template.jinja @@ -0,0 +1,47 @@ +{{ bos_token }} +{%- if messages[0]['role'] == 'system' -%} + {%- if messages[0]['content'] is string -%} + {%- set first_user_prefix = messages[0]['content'] + ' + +' -%} + {%- else -%} + {%- set first_user_prefix = messages[0]['content'][0]['text'] + ' + +' -%} + {%- endif -%} + {%- set loop_messages = messages[1:] -%} +{%- else -%} + {%- set first_user_prefix = "" -%} + {%- set loop_messages = messages -%} +{%- endif -%} +{%- for message in loop_messages -%} + {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%} + {{ raise_exception("Conversation roles must alternate user/assistant/user/assistant/...") }} + {%- endif -%} + {%- if (message['role'] == 'assistant') -%} + {%- set role = "model" -%} + {%- else -%} + {%- set role = message['role'] -%} + {%- endif -%} + {{ '' + role + ' +' + (first_user_prefix if loop.first else "") }} + {%- if message['content'] is string -%} + {{ message['content'] | trim }} + {%- elif message['content'] is iterable -%} + {%- for item in message['content'] -%} + {%- if item['type'] == 'image' -%} + {{ '' }} + {%- elif item['type'] == 'text' -%} + {{ item['text'] | trim }} + {%- endif -%} + {%- endfor -%} + {%- else -%} + {{ raise_exception("Invalid content type") }} + {%- endif -%} + {{ ' +' }} +{%- endfor -%} +{%- if add_generation_prompt -%} + {{'model +'}} +{%- endif -%} diff --git a/config.json b/config.json new file mode 100644 index 0000000..f239cc9 --- /dev/null +++ b/config.json @@ -0,0 +1,62 @@ +{ + "_sliding_window_pattern": 6, + "architectures": [ + "Gemma3ForCausalLM" + ], + "attention_bias": false, + "attention_dropout": 0.0, + "attn_logit_softcapping": null, + "bos_token_id": 2, + "dtype": "bfloat16", + "eos_token_id": 1, + "final_logit_softcapping": null, + "head_dim": 256, + "hidden_activation": "gelu_pytorch_tanh", + "hidden_size": 640, + "initializer_range": 0.02, + "intermediate_size": 2048, + "layer_types": [ + "sliding_attention", + "sliding_attention", + "sliding_attention", + "sliding_attention", + "sliding_attention", + "full_attention", + "sliding_attention", + "sliding_attention", + "sliding_attention", + "sliding_attention", + "sliding_attention", + "full_attention", + "sliding_attention", + "sliding_attention", + "sliding_attention", + "sliding_attention", + "sliding_attention", + "full_attention" + ], + "max_position_embeddings": 32768, + "model_type": "gemma3_text", + "num_attention_heads": 4, + "num_hidden_layers": 18, + "num_key_value_heads": 1, + "pad_token_id": 0, + "query_pre_attn_scalar": 256, + "rms_norm_eps": 1e-06, + "rope_parameters": { + "full_attention": { + "rope_theta": 1000000.0, + "rope_type": "default" + }, + "sliding_attention": { + "rope_theta": 10000.0, + "rope_type": "default" + } + }, + "sliding_window": 512, + "tie_word_embeddings": true, + "transformers_version": "5.14.1", + "use_bidirectional_attention": false, + "use_cache": true, + "vocab_size": 262144 +} diff --git a/generation_config.json b/generation_config.json new file mode 100644 index 0000000..d87672b --- /dev/null +++ b/generation_config.json @@ -0,0 +1,11 @@ +{ + "cache_implementation": "hybrid", + "do_sample": true, + "eos_token_id": [ + 1, + 106 + ], + "top_k": 64, + "top_p": 0.95, + "transformers_version": "5.14.1" +} diff --git a/lora-adapter/README.md b/lora-adapter/README.md new file mode 100644 index 0000000..6eee7bc --- /dev/null +++ b/lora-adapter/README.md @@ -0,0 +1,209 @@ +--- +base_model: google/gemma-3-270m-it +library_name: peft +pipeline_tag: text-generation +tags: +- base_model:adapter:google/gemma-3-270m-it +- lora +- sft +- transformers +- trl +--- + +# Model Card for Model ID + + + + + +## Model Details + +### Model Description + + + + + +- **Developed by:** [More Information Needed] +- **Funded by [optional]:** [More Information Needed] +- **Shared by [optional]:** [More Information Needed] +- **Model type:** [More Information Needed] +- **Language(s) (NLP):** [More Information Needed] +- **License:** [More Information Needed] +- **Finetuned from model [optional]:** [More Information Needed] + +### Model Sources [optional] + + + +- **Repository:** [More Information Needed] +- **Paper [optional]:** [More Information Needed] +- **Demo [optional]:** [More Information Needed] + +## Uses + + + +### Direct Use + + + +[More Information Needed] + +### Downstream Use [optional] + + + +[More Information Needed] + +### Out-of-Scope Use + + + +[More Information Needed] + +## Bias, Risks, and Limitations + + + +[More Information Needed] + +### Recommendations + + + +Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations. + +## How to Get Started with the Model + +Use the code below to get started with the model. + +[More Information Needed] + +## Training Details + +### Training Data + + + +[More Information Needed] + +### Training Procedure + + + +#### Preprocessing [optional] + +[More Information Needed] + + +#### Training Hyperparameters + +- **Training regime:** [More Information Needed] + +#### Speeds, Sizes, Times [optional] + + + +[More Information Needed] + +## Evaluation + + + +### Testing Data, Factors & Metrics + +#### Testing Data + + + +[More Information Needed] + +#### Factors + + + +[More Information Needed] + +#### Metrics + + + +[More Information Needed] + +### Results + +[More Information Needed] + +#### Summary + + + +## Model Examination [optional] + + + +[More Information Needed] + +## Environmental Impact + + + +Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700). + +- **Hardware Type:** [More Information Needed] +- **Hours used:** [More Information Needed] +- **Cloud Provider:** [More Information Needed] +- **Compute Region:** [More Information Needed] +- **Carbon Emitted:** [More Information Needed] + +## Technical Specifications [optional] + +### Model Architecture and Objective + +[More Information Needed] + +### Compute Infrastructure + +[More Information Needed] + +#### Hardware + +[More Information Needed] + +#### Software + +[More Information Needed] + +## Citation [optional] + + + +**BibTeX:** + +[More Information Needed] + +**APA:** + +[More Information Needed] + +## Glossary [optional] + + + +[More Information Needed] + +## More Information [optional] + +[More Information Needed] + +## Model Card Authors [optional] + +[More Information Needed] + +## Model Card Contact + +[More Information Needed] +### Framework versions + +- PEFT 0.19.1 \ No newline at end of file diff --git a/lora-adapter/adapter_config.json b/lora-adapter/adapter_config.json new file mode 100644 index 0000000..73557b3 --- /dev/null +++ b/lora-adapter/adapter_config.json @@ -0,0 +1,43 @@ +{ + "alora_invocation_tokens": null, + "alpha_pattern": {}, + "arrow_config": null, + "auto_mapping": null, + "base_model_name_or_path": "google/gemma-3-270m-it", + "bias": "none", + "corda_config": null, + "ensure_weight_tying": false, + "eva_config": null, + "exclude_modules": null, + "fan_in_fan_out": false, + "inference_mode": true, + "init_lora_weights": true, + "layer_replication": null, + "layers_pattern": null, + "layers_to_transform": null, + "loftq_config": {}, + "lora_alpha": 16, + "lora_bias": false, + "lora_dropout": 0.05, + "lora_ga_config": null, + "megatron_config": null, + "megatron_core": "megatron.core", + "modules_to_save": null, + "peft_type": "LORA", + "peft_version": "0.19.1", + "qalora_group_size": 16, + "r": 8, + "rank_pattern": {}, + "revision": null, + "target_modules": [ + "v_proj", + "q_proj" + ], + "target_parameters": null, + "task_type": "CAUSAL_LM", + "trainable_token_indices": null, + "use_bdlora": null, + "use_dora": false, + "use_qalora": false, + "use_rslora": false +} \ No newline at end of file diff --git a/lora-adapter/adapter_model.safetensors b/lora-adapter/adapter_model.safetensors new file mode 100644 index 0000000..9b42d8d --- /dev/null +++ b/lora-adapter/adapter_model.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b280e8f06193a11980c5a1f4d4f616bc384feb5c20a7ac4120d5a00f7c0d73d9 +size 1483904 diff --git a/lora-adapter/chat_template.jinja b/lora-adapter/chat_template.jinja new file mode 100644 index 0000000..1117055 --- /dev/null +++ b/lora-adapter/chat_template.jinja @@ -0,0 +1,47 @@ +{{ bos_token }} +{%- if messages[0]['role'] == 'system' -%} + {%- if messages[0]['content'] is string -%} + {%- set first_user_prefix = messages[0]['content'] + ' + +' -%} + {%- else -%} + {%- set first_user_prefix = messages[0]['content'][0]['text'] + ' + +' -%} + {%- endif -%} + {%- set loop_messages = messages[1:] -%} +{%- else -%} + {%- set first_user_prefix = "" -%} + {%- set loop_messages = messages -%} +{%- endif -%} +{%- for message in loop_messages -%} + {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%} + {{ raise_exception("Conversation roles must alternate user/assistant/user/assistant/...") }} + {%- endif -%} + {%- if (message['role'] == 'assistant') -%} + {%- set role = "model" -%} + {%- else -%} + {%- set role = message['role'] -%} + {%- endif -%} + {{ '' + role + ' +' + (first_user_prefix if loop.first else "") }} + {%- if message['content'] is string -%} + {{ message['content'] | trim }} + {%- elif message['content'] is iterable -%} + {%- for item in message['content'] -%} + {%- if item['type'] == 'image' -%} + {{ '' }} + {%- elif item['type'] == 'text' -%} + {{ item['text'] | trim }} + {%- endif -%} + {%- endfor -%} + {%- else -%} + {{ raise_exception("Invalid content type") }} + {%- endif -%} + {{ ' +' }} +{%- endfor -%} +{%- if add_generation_prompt -%} + {{'model +'}} +{%- endif -%} diff --git a/lora-adapter/tokenizer.json b/lora-adapter/tokenizer.json new file mode 100644 index 0000000..f74d183 --- /dev/null +++ b/lora-adapter/tokenizer.json @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:daab2354f8a74e70d70b4d1f804939b68a8c9624dd06cb7858e52dd8970e9726 +size 33384567 diff --git a/lora-adapter/tokenizer_config.json b/lora-adapter/tokenizer_config.json new file mode 100644 index 0000000..60cdfef --- /dev/null +++ b/lora-adapter/tokenizer_config.json @@ -0,0 +1,25 @@ +{ + "backend": "tokenizers", + "boi_token": "", + "bos_token": "", + "clean_up_tokenization_spaces": false, + "eoi_token": "", + "eos_token": "", + "image_token": "", + "is_local": false, + "local_files_only": false, + "mask_token": "", + "model_max_length": 1000000000000000019884624838656, + "model_specific_special_tokens": { + "boi_token": "", + "eoi_token": "", + "image_token": "" + }, + "pad_token": "", + "padding_side": "left", + "sp_model_kwargs": null, + "spaces_between_special_tokens": false, + "tokenizer_class": "GemmaTokenizer", + "unk_token": "", + "use_default_system_prompt": false +} diff --git a/model.safetensors b/model.safetensors new file mode 100644 index 0000000..424e913 --- /dev/null +++ b/model.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:94b78fdb4dfa39ba7cda48d788d3ef054f916228b7bcbbbc748caae29121f042 +size 536223056 diff --git a/tokenizer.json b/tokenizer.json new file mode 100644 index 0000000..f74d183 --- /dev/null +++ b/tokenizer.json @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:daab2354f8a74e70d70b4d1f804939b68a8c9624dd06cb7858e52dd8970e9726 +size 33384567 diff --git a/tokenizer_config.json b/tokenizer_config.json new file mode 100644 index 0000000..60cdfef --- /dev/null +++ b/tokenizer_config.json @@ -0,0 +1,25 @@ +{ + "backend": "tokenizers", + "boi_token": "", + "bos_token": "", + "clean_up_tokenization_spaces": false, + "eoi_token": "", + "eos_token": "", + "image_token": "", + "is_local": false, + "local_files_only": false, + "mask_token": "", + "model_max_length": 1000000000000000019884624838656, + "model_specific_special_tokens": { + "boi_token": "", + "eoi_token": "", + "image_token": "" + }, + "pad_token": "", + "padding_side": "left", + "sp_model_kwargs": null, + "spaces_between_special_tokens": false, + "tokenizer_class": "GemmaTokenizer", + "unk_token": "", + "use_default_system_prompt": false +}