31 lines
999 B
Markdown
31 lines
999 B
Markdown
---
|
||
library_name: transformers
|
||
base_model:
|
||
- google/gemma-3-1b-it
|
||
tags:
|
||
- gemma3
|
||
- math
|
||
- merged
|
||
license: gemma
|
||
---
|
||
|
||
# Gemma 3 1B IT — MetaMathQA Merged (α=0.5)
|
||
|
||
A merged model created by interpolating the weights of a MetaMathQA-finetuned Gemma 3 1B IT with the original base model.
|
||
|
||
## Method
|
||
|
||
1. **Fine-tune** `google/gemma-3-1b-it` on 7,000 samples from [MetaMathQA](https://huggingface.co/datasets/meta-math/MetaMathQA) using SFT.
|
||
2. **Merge** the fine-tuned weights back into the base model via linear interpolation with α=0.5:
|
||
|
||
$$\theta_{\text{merged}} = \alpha \cdot \theta_{\text{FT}} + (1 - \alpha) \cdot \theta_{\text{base}}$$
|
||
|
||
This simple averaging actually improves task-specific gain from fine-tuning while retaining more of the base model's instruction following that pure FT degrades.
|
||
|
||
## Results
|
||
|
||
| Method | MMLU Redux | GSM8K | IFEval |
|
||
|---|---|---|---|
|
||
| Base | 39.79 | 33.66 | **40.48** |
|
||
| FT | **41.02** | 37.15 | 28.84 |
|
||
| **Merged** | 40.53 | **39.58** | 36.41 | |