131 lines
8.2 KiB
Markdown
131 lines
8.2 KiB
Markdown
---
|
|
license: apache-2.0
|
|
language:
|
|
- en
|
|
pipeline_tag: text-generation
|
|
tags:
|
|
- unsloth
|
|
- qwen
|
|
- qwen2.5
|
|
- math
|
|
- reasoning
|
|
- alpaca
|
|
- pytorch
|
|
- custom-finetune
|
|
- lora-merged
|
|
base_model: unsloth/Qwen2.5-Math-1.5B
|
|
datasets:
|
|
- Xerv-AI/GRAD
|
|
- yahma/alpaca-cleaned
|
|
inference:
|
|
parameters:
|
|
repetition_penalty: 1.15
|
|
max_new_tokens: 256
|
|
temperature: 0.5
|
|
examples:
|
|
- text: "### Instruction:\nProvide a step-by-step logical proof finding the eigenvalues of the matrix [[2, 1], [1, 2]].\n### Response:\n"
|
|
|
|
widget:
|
|
- example_title: Fibonacci (Python)
|
|
messages:
|
|
- role: system
|
|
content: You are a chatbot who can help code!
|
|
- role: user
|
|
content: Write me a function to calculate the first 10 digits of the fibonacci sequence in Python and print it out to the CLI.
|
|
---
|
|
|
|
|
|
## 🌌 Xerv-AI/Ada: The Multi-Modal Mathematical Generalist SLM
|
|
**Ada** is an ultra-lightweight, high-speed, and highly optimized reasoning Small Language Model (SLM) derived from the powerful **Qwen2.5-Math-1.5B** architecture. Engineered specifically to bridge the gap between hyper-specialized graduate-level mathematical proofs and standard conversational utility, Ada solves the notorious "catastrophic forgetting" problem often found in math-heavy fine-tunes.
|
|
Whether you need a step-by-step calculus breakdown, a topological proof in LaTeX, or just a simple conversational assistant for daily tasks, Ada delivers state-of-the-art performance for a 1.5 Billion parameter model.
|
|
|
|
### 🚀 Model Overview
|
|
Standard math-specific LLMs frequently suffer from domain overfitting. When prompted with basic conversational queries, they either hallucinate lengthy pseudo-proofs or fail entirely to understand the user's intent. **Xerv-AI/Ada** was meticulously engineered to resolve this by utilizing a carefully balanced, dual-distribution training dataset, allowing it to act as both a rigorous STEM assistant and a general-purpose chat model.
|
|
|
|
| Specification | Details |
|
|
| :--- | :--- |
|
|
| **Model Name** | Xerv-AI/Ada |
|
|
| **Base Architecture** | unsloth/Qwen2.5-Math-1.5B |
|
|
| **Parameter Count** | 1.5 Billion |
|
|
| **Primary Capabilities** | Graduate-level STEM reasoning, logical deduction, and mathematical proofs. |
|
|
| **Secondary Capabilities** | General conversational instruction-following, roleplay, and basic coding. |
|
|
| **Training Framework** | QLoRA via Unsloth (Triton kernels). |
|
|
| **Precision** | Merged 16-bit (Fine-tuned in 4-bit). |
|
|
| **License** | Apache-2.0 | <br> ### 🔬 Core Capabilities & Strengths <br> * **Balanced Generalization:** Ada seamlessly transitions between casual conversation and intense analytical problem-solving without format-forced hallucinations. <br> * **Advanced STEM Reasoning:** Fully optimized to generate detailed, multi-step logical proofs in advanced algebra, calculus, topology, and physics. <br> * **Hardware Optimized for Edge Deployment:** Designed to run at maximum inference throughput on low-VRAM consumer hardware (such as a single 16GB NVIDIA T4 GPU, Mac M-series chips, or edge devices) using 4-bit quantization. <br> * **Impeccable Formatting:** Native understanding of structural formatting, easily outputting highly readable markdown and structured logic steps. <br> ### 🏗 Architecture & Training Methodology <br> Ada was trained using Supervised Fine-Tuning (SFT) targeting the attention mechanisms of the base model. Utilizing **Unsloth** on a standard Google Colab NVIDIA T4 GPU, the training leveraged Low-Rank Adaptation (LoRA) to maximize efficiency before being merged into a standalone 16-bit Hugging Face model. <br> * **Target Modules:** q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj <br> * **LoRA Rank (r):** 16 <br> * **LoRA Alpha:** 16 <br> * **Optimizer:** adamw_8bit <br> * **Learning Rate:** 2e-4 <br> * **Effective Batch Size:** 8 (Batch size 2 with 4 Gradient Accumulation steps) <br> ### 📚 The Dataset: Dual-Distribution Blending <br> To achieve generalization and prevent catastrophic forgetting, Ada was fine-tuned on a strict 50/50 blend of two distinct datasets, batched and streamed via high-throughput Parquet files:
|
|
| Dataset | Sample Size | Description & Purpose |
|
|
| :--- | :--- | :--- |
|
|
| **Xerv-AI/GRAD** | ~1.93k rows | A proprietary synthetic dataset containing exceptionally long (average 8,000 characters) graduate and research-level mathematical proofs. This instills deep reasoning and strict formatting. |
|
|
| **yahma/alpaca-cleaned** | ~2.00k rows | A refined subset of the standard Alpaca dataset. This teaches the model conversational flow, roleplay, basic Q&A, and crucially, *when not to use complex math*. |
|
|
|
|
### 💻 Usage & Python Inference Guide
|
|
The model is highly responsive to the standard **Alpaca Instruction/Response template**.
|
|
**Important Inference Note:** For best results, use a repetition_penalty of roughly **1.15**. This acts as a crucial guardrail to prevent the model from infinitely looping through mathematical steps on overly simple arithmetic queries.
|
|
**1. Installation Requirements**
|
|
```bash
|
|
pip install unsloth transformers accelerate torch
|
|
```
|
|
**2. Fast Inference Script**
|
|
```python
|
|
from unsloth import FastLanguageModel
|
|
import torch
|
|
# Configuration
|
|
repo_name = "Xerv-AI/Ada"
|
|
max_seq_length = 2048
|
|
# Load the model and tokenizer (4-bit recommended for low-VRAM)
|
|
model, tokenizer = FastLanguageModel.from_pretrained(
|
|
model_name = repo_name,
|
|
max_seq_length = max_seq_length,
|
|
dtype = None,
|
|
load_in_4bit = True,
|
|
)
|
|
# Enable optimized inference mode
|
|
FastLanguageModel.for_inference(model)
|
|
# Define the universal prompt template
|
|
universal_prompt = """### Instruction:
|
|
{}
|
|
### Response:
|
|
{}"""
|
|
# Prepare your query
|
|
query = "Provide a step-by-step logical proof finding the eigenvalues of the matrix [[2, 1], [1, 2]]."
|
|
inputs = tokenizer(
|
|
[universal_prompt.format(query, "")],
|
|
return_tensors = "pt"
|
|
).to("cuda")
|
|
print("Generating analytical response...")
|
|
# Generate the output
|
|
outputs = model.generate(
|
|
**inputs,
|
|
max_new_tokens = 1024,
|
|
max_length = None,
|
|
use_cache = True,
|
|
repetition_penalty = 1.15, # Critical: prevents generation loops
|
|
pad_token_id = tokenizer.eos_token_id
|
|
)
|
|
# Decode and print the result
|
|
response = tokenizer.batch_decode(outputs, skip_special_tokens = True)[0]
|
|
print(f"\n{'='*50}\nOutput:\n{'='*50}")
|
|
print(response.split("### Response:\n")[-1])
|
|
```
|
|
|
|
### Performance Summary
|
|
|
|
| Dataset | Accuracy |
|
|
| :--- | :--- |
|
|
| **GSM8K** | **40.00%** |
|
|
| **MATH** |**60.00%** |
|
|
| **MATH-Hard** |**50.00%** |
|
|
| **GRAD** |**40.00%** |
|
|
|
|
### 🛡️ Safety & Alignment Guardrails
|
|
Despite being fine-tuned on raw mathematical logic and conversational instruction data, Ada successfully retains its foundational safety alignments. Because only 1% to 2% of the parameters were actively updated via LoRA (and subsequently merged), the original base Qwen2.5 weights responsible for safety remain fully intact.
|
|
* **Content Moderation:** The model actively refuses to generate explicit, illegal, or harmful content, relying on the RLHF and DPO safety guardrails instilled during Alibaba's original pre-training phase.
|
|
### ⚠️ Limitations & Known Biases
|
|
While Ada punches well above its 1.5B weight class, it is important to acknowledge the limitations inherent to Small Language Models:
|
|
* **Arithmetic Hallucinations:** Ada is exceptionally capable at symbolic logic, structural breakdowns, and mathematical theory. However, like many SLMs, it can occasionally suffer from minor arithmetic errors (e.g., basic addition/subtraction mistakes) deep within multi-page proofs. Always verify raw calculations.
|
|
* **Language Constraint:** The model is optimized exclusively for **English** text and standard mathematical notation.
|
|
* **Prompt Sensitivity:** Ada performs at its absolute peak when math queries explicitly ask for a "proof," "step-by-step breakdown," or "logical analysis" within the instruction block.
|
|
* **World Knowledge:** It lacks the broad, encyclopedic trivia knowledge found in massive 70B+ parameter models.
|
|
### 🤝 Acknowledgements
|
|
* **Alibaba Cloud:** For the phenomenal, state-of-the-art base Qwen2.5-Math architecture.
|
|
* **Unsloth AI:** For the Triton-optimized training kernels that made compiling and fine-tuning this model possible and highly efficient on consumer hardware.
|
|
* **Xerv-AI:** For the curation of the GRAD synthetic dataset powering the advanced reasoning capabilities. |