29 lines
892 B
Markdown
29 lines
892 B
Markdown
---
|
|
license: apache-2.0
|
|
---
|
|
|
|
# mistral-7b-metamathqa-sft
|
|
|
|
mistral-7b-metamathqa-sft is an SFT fine-tuned version of [unsloth/mistral-7b-bnb-4bit](https://huggingface.co/unsloth/mistral-7b-bnb-4bit) using the [meta-math/MetaMathQA](https://huggingface.co/datasets/meta-math/MetaMathQA) dataset.
|
|
|
|
## Fine-tuning configuration
|
|
### LoRA
|
|
- r: 256
|
|
- LoRA alpha: 128
|
|
- LoRA dropout: 0.0
|
|
|
|
### Training arguments
|
|
- Epochs: 1
|
|
- Batch size: 4
|
|
- Gradient accumulation steps: 6
|
|
- Optimizer: adamw_torch_fused
|
|
- Max steps: 100
|
|
- Learning rate: 0.0002
|
|
- Weight decay: 0.1
|
|
- Learning rate scheduler type: linear
|
|
- Max seq length: 2048
|
|
- 4-bit bnb: True
|
|
|
|
Trained with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.
|
|
|
|
[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth) |