Llama-3.2-1B-MathCodeInstruct-5k/README.md

license, license_name, license_link, base_model, tags, datasets, language, library_name, pipeline_tag, model-index

license	license_name	license_link	base_model	tags	datasets	language	library_name	pipeline_tag	model-index
other	llama3.2	https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/LICENSE	unsloth/Llama-3.2-1B	math fine-tuned lora unsloth llama	MathLLMs/MathCodeInstruct	en	transformers	text-generation	name results Llama-3.2-1B-MathCodeInstruct-{{SIZE}} task dataset metrics type name text-generation GSM8K type name gsm8k GSM8K type value name exact_match GSM8K_ACC exact match (flexible-extract, 5-shot) task dataset metrics type name text-generation ARC-Challenge type name ai2_arc ARC-Challenge type value name acc_norm ARC_ACC acc_norm (25-shot) task dataset metrics type name text-generation HellaSwag type name hellaswag HellaSwag type value name acc_norm HELLASWAG_ACC acc_norm (10-shot) task dataset metrics type name text-generation WinoGrande type name winogrande WinoGrande type value name acc WINOGRANDE_ACC acc (5-shot) task dataset metrics type name text-generation MMLU type name mmlu MMLU type value name acc MMLU_ACC acc (5-shot)

Llama-3.2-1B-MathCodeInstruct-5k

A Llama-3.2-1B fine-tune on 5k examples from MathLLMs/MathCodeInstruct, trained to solve math word problems with step-by-step natural-language reasoning interleaved with executable Python.

This is one of three sibling models trained on {5k, 10k, 20k}-example subsets of the same dataset, to study how fine-tuning data volume trades off against both math performance and general capability. See the training write-up for the full comparison across all three.

Training details

Base model	unsloth/Llama-3.2-1B
Method	LoRA (r=16, α=16, dropout=0) on all attention + MLP projections, merged to full weights
Dataset	MathLLMs/MathCodeInstruct, 5k training examples
Epochs	1
Effective batch size	16 (batch 1 × grad. accum. 16)
Learning rate	2e-4, cosine schedule, warmup ratio 0.03
Hardware	1× RTX 4060 (8GB)
Framework	Unsloth + TRL SFTTrainer

Benchmark results

All benchmarks run with lm-evaluation-harness, each at its standard published shot count, compared against the un-tuned base model.

Benchmark	Llama-3.2-1B (base)	MathCodeInstruct-5k	Change
GSM8K	5.8%	7.4%	🟢 +1.5%
ARC-Challenge	36.9%	36.8%	⚪ -0.1%
HellaSwag	64.2%	63.8%	🔴 -0.3%
WinoGrande	60.8%	62.4%	🟢 +1.7%

Speed: 40.67 tokens/sec (base model: 40.59 tokens/sec)

MMLU by category

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "OliverSundaram/Llama-3.2-1B-MathCodeInstruct-5k"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16", device_map="auto")

messages = [
    {"role": "system", "content": "Below is a math problem. Please solve it step by step."},
    {"role": "user", "content": "If a train travels 60 miles in 45 minutes, what is its speed in miles per hour?"},
]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
output = model.generate(inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Limitations

• Trained on a single epoch of a 5k-example subset — not intended to be a general-purpose assistant.
• MMLU/ARC/HellaSwag/WinoGrande scores reflect a small 1B-parameter base model and should be read relative to the base model's own scores, not against much larger models.
• No safety alignment or RLHF was applied beyond what the base Llama-3.2-1B already has.