35 lines
972 B
Markdown
35 lines
972 B
Markdown
|
---
|
||
|
|
license: apache-2.0
|
||
|
|
tags:
|
||
|
|
- sft
|
||
|
|
dataset:
|
||
|
|
- teknium/openhermes
|
||
|
|
base_model:
|
||
|
|
- unsloth/mistral-7b-bnb-4bit
|
||
|
|
---
|
||
|
|
|
||
|
|
# mistral-7b-openhermes-sft
|
||
|
|
|
||
|
|
mistral-7b-openhermes-sft is an SFT fine-tuned version of [unsloth/mistral-7b-bnb-4bit](https://huggingface.co/unsloth/mistral-7b-bnb-4bit) using the [teknium/openhermes](https://huggingface.co/datasets/teknium/openhermes) dataset.
|
||
|
|
|
||
|
|
## Fine-tuning configuration
|
||
|
|
### LoRA
|
||
|
|
- r: 256
|
||
|
|
- LoRA alpha: 128
|
||
|
|
- LoRA dropout: 0.0
|
||
|
|
|
||
|
|
### Training arguments
|
||
|
|
- Epochs: 1
|
||
|
|
- Batch size: 4
|
||
|
|
- Gradient accumulation steps: 6
|
||
|
|
- Optimizer: adamw_torch_fused
|
||
|
|
- Max steps: 100
|
||
|
|
- Learning rate: 0.0002
|
||
|
|
- Weight decay: 0.1
|
||
|
|
- Learning rate scheduler type: linear
|
||
|
|
- Max seq length: 2048
|
||
|
|
- 4-bit bnb: True
|
||
|
|
|
||
|
|
Trained with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.
|
||
|
|
|
||
|
|
[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)
|