license, base_model, base_model_relation, tags, language, datasets

license	base_model	base_model_relation	tags	language	datasets
apache-2.0	HuggingFaceTB/SmolLM3-3B-Base	finetune	smollm3 fine-tuned lora math conversational text-generation-inference	en	openai/gsm8k

Al-Khwarizmi-3B

An AI math tutor named after Muhammad al-Khwarizmi, the 9th-century mathematician whose name is the origin of the word "algorithm".

Fine-tune of HuggingFaceTB/SmolLM3-3B-Base, trained in two stages — full fine-tuning followed by LoRA — to solve grade-school math word problems with clear, step-by-step reasoning.

Highlights

• 87.21% mean token accuracy on held-out validation data — up from 84.75% after the initial full fine-tune, and 82.4% at the very first checkpoint
• Validation loss reduced by ~19% across the full pipeline (0.678 → 0.462), with training and validation loss tracking closely throughout every stage — no overfitting observed
• Trained on the complete GSM8K dataset (both main and socratic reasoning styles) across two LoRA passes, on top of an initial full fine-tune
• Available in this repo as safetensors. GGUF version with quantization (BF16 and Q8_0) for efficient usage on CPU and a smaller size available here.

Training Details

Stage 1 — Full fine-tuning

Dataset	GSM8K (main), 1,000 random samples, 90/10 train/val split
Steps	450 (1 epoch)
Learning rate	5e-5, cosine schedule
Final validation loss / accuracy	0.569 / 84.75%

Stage 2 — LoRA fine-tuning

Method	LoRA, r=16, all-linear target modules
Dataset	Full GSM8K — both main and socratic reasoning styles
Steps	3,550 (combined across two passes)
Learning rate	5e-5, cosine schedule
Final validation loss / accuracy	0.462 / 87.21%

Run as two consecutive passes over the dataset, with the main/socratic split swapped between them so every problem was seen in both reasoning styles.

Limitations

Fine-tuned primarily on GSM8K-style problems (single correct numeric answer, grade-school arithmetic/word problems) — performance on more complex, multi-part, or differently-structured math problems is untested. Occasional arithmetic slips on multi-step problems can still occur, consistent with known limitations of models at this scale.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "mzoelfakar/Al-Khwarizmi-3B"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, dtype=torch.bfloat16, device_map="auto")

messages = [
    {"role": "system", "content": "You are a math tutor. Solve problems step by step."},
    {"role": "user", "content": "If a train travels 120 miles in 2 hours, what is its average speed?"}
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=300, temperature=0.7, do_sample=True)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Note on raw output formatting

Because this model was fine-tuned on GSM8K (including the socratic reasoning style), raw generations may contain training artifacts not meant for direct display:

• <<...>> — calculator-style intermediate annotations
• ** — separator between a sub-question and its calculation in Socratic-style reasoning (not Markdown bold)
• #### <answer> — marker preceding the final numeric answer
• * — used as a multiplication sign (e.g. 8*9); if two or more appear in the same response, Markdown may pair them as emphasis delimiters, causing text between them to render in italic with the asterisks hidden

If you're piping output through a Markdown renderer or displaying it in a UI, you'll likely want to strip or reformat these first, since ** in particular can be misread as Markdown bold syntax if left unescaped.

Try it online

A live chat demo is available via Colab: Al-Khwarizmi-3B.ipynb

Credits

Fine-tuned by Mohamed Zoelfakar, as part of Hugging Face's smol-course.