28b628771a
Model: Raghav-Singhal/pretrain-normal-smollm-1p7b-100B-20n-2048sl-960gbsz-sft-tulu3sft Source: Original Platform
33 lines
915 B
Markdown
33 lines
915 B
Markdown
---
|
|
library_name: transformers
|
|
pipeline_tag: text-generation
|
|
tags:
|
|
- llama
|
|
- causal-lm
|
|
- instruction-tuned
|
|
- bfloat16
|
|
---
|
|
|
|
# pretrain-normal-smollm-1p7b-100B-20n-2048sl-960gbsz-sft-tulu3sft
|
|
|
|
Supervised fine-tuned Hugging Face checkpoint from the Model Raising post-training run on `allenai/tulu-3-sft-mixture`.
|
|
|
|
## Details
|
|
|
|
- Base model: `pretrain-normal-smollm-1p7b-100B-20n-2048sl-960gbsz`
|
|
- Architecture: `LlamaForCausalLM`
|
|
- Precision on disk: `bfloat16`
|
|
- Tokenizer: `HuggingFaceTB/SmolLM2-1.7B-Instruct`
|
|
- Training data: `allenai/tulu-3-sft-mixture`
|
|
- Sequence length: `2048`
|
|
- Global batch size: `128`
|
|
- Learning rate schedule: cosine decay with `lr=1e-5`, `warmup_ratio=0.03`, `min_lr_ratio=0.1`
|
|
|
|
## Training Summary
|
|
|
|
- Final train loss: `0.9213777665`
|
|
- Train runtime: `7286.4s`
|
|
- Train steps: `7339`
|
|
|
|
This repo contains the final saved SFT checkpoint from `/posttraining/checkpoints/sft-tulu3-4n-gbs128`.
|