---
library_name: transformers
pipeline_tag: text-generation
tags:
- llama
- causal-lm
- instruction-tuned
- bfloat16
---

# pretrain-normal-smollm-1p7b-100B-20n-2048sl-960gbsz-sft-tulu3sft

Supervised fine-tuned Hugging Face checkpoint from the Model Raising post-training run on `allenai/tulu-3-sft-mixture`.

## Details

- Base model: `pretrain-normal-smollm-1p7b-100B-20n-2048sl-960gbsz`
- Architecture: `LlamaForCausalLM`
- Precision on disk: `bfloat16`
- Tokenizer: `HuggingFaceTB/SmolLM2-1.7B-Instruct`
- Training data: `allenai/tulu-3-sft-mixture`
- Sequence length: `2048`
- Global batch size: `128`
- Learning rate schedule: cosine decay with `lr=1e-5`, `warmup_ratio=0.03`, `min_lr_ratio=0.1`

## Training Summary

- Final train loss: `0.9213777665`
- Train runtime: `7286.4s`
- Train steps: `7339`

This repo contains the final saved SFT checkpoint from `/posttraining/checkpoints/sft-tulu3-4n-gbs128`.