24 lines
592 B
Markdown
24 lines
592 B
Markdown
---
|
|
license: apache-2.0
|
|
---
|
|
|
|
# gemma-7b-ultrachat-sft
|
|
|
|
gemma-7b-ultrachat-sft is an SFT fine-tuned version of [google/gemma-7b](https://huggingface.co/google/gemma-7b) using the [stingning/ultrachat](https://huggingface.co/datasets/stingning/ultrachat) dataset.
|
|
|
|
## Fine-tuning configuration
|
|
### LoRA
|
|
- LoRA r: 8
|
|
- LoRA alpha: 16
|
|
- LoRA dropout: 0.1
|
|
|
|
### Training arguments
|
|
- Epochs: 1
|
|
- Batch size: 4
|
|
- Gradient accumulation steps: 6
|
|
- Optimizer: paged_adamw_32bit
|
|
- Max steps: 100
|
|
- Learning rate: 0.0002
|
|
- Weight decay: 0.001
|
|
- Learning rate scheduler type: constant
|
|
- Max seq length: 2048 |