初始化项目,由ModelHub XC社区提供模型
Model: locailabs/gemma-3-1b-it-sft-metamathqa-modelmerge Source: Original Platform
This commit is contained in:
31
README.md
Normal file
31
README.md
Normal file
@@ -0,0 +1,31 @@
|
||||
---
|
||||
library_name: transformers
|
||||
base_model:
|
||||
- google/gemma-3-1b-it
|
||||
tags:
|
||||
- gemma3
|
||||
- math
|
||||
- merged
|
||||
license: gemma
|
||||
---
|
||||
|
||||
# Gemma 3 1B IT — MetaMathQA Merged (α=0.5)
|
||||
|
||||
A merged model created by interpolating the weights of a MetaMathQA-finetuned Gemma 3 1B IT with the original base model.
|
||||
|
||||
## Method
|
||||
|
||||
1. **Fine-tune** `google/gemma-3-1b-it` on 7,000 samples from [MetaMathQA](https://huggingface.co/datasets/meta-math/MetaMathQA) using SFT.
|
||||
2. **Merge** the fine-tuned weights back into the base model via linear interpolation with α=0.5:
|
||||
|
||||
$$\theta_{\text{merged}} = \alpha \cdot \theta_{\text{FT}} + (1 - \alpha) \cdot \theta_{\text{base}}$$
|
||||
|
||||
This simple averaging actually improves task-specific gain from fine-tuning while retaining more of the base model's instruction following that pure FT degrades.
|
||||
|
||||
## Results
|
||||
|
||||
| Method | MMLU Redux | GSM8K | IFEval |
|
||||
|---|---|---|---|
|
||||
| Base | 39.79 | 33.66 | **40.48** |
|
||||
| FT | **41.02** | 37.15 | 28.84 |
|
||||
| **Merged** | 40.53 | **39.58** | 36.41 |
|
||||
Reference in New Issue
Block a user