library_name, pipeline_tag, tags

library_name	pipeline_tag	tags
transformers	text-generation	llama causal-lm instruction-tuned bfloat16

pretrain-normal-smollm-1p7b-100B-20n-2048sl-960gbsz-sft-tulu3sft

Supervised fine-tuned Hugging Face checkpoint from the Model Raising post-training run on allenai/tulu-3-sft-mixture.

Details

• Base model: pretrain-normal-smollm-1p7b-100B-20n-2048sl-960gbsz
• Architecture: LlamaForCausalLM
• Precision on disk: bfloat16
• Tokenizer: HuggingFaceTB/SmolLM2-1.7B-Instruct
• Training data: allenai/tulu-3-sft-mixture
• Sequence length: 2048
• Global batch size: 128
• Learning rate schedule: cosine decay with lr=1e-5, warmup_ratio=0.03, min_lr_ratio=0.1

Training Summary

• Final train loss: 0.9213777665
• Train runtime: 7286.4s
• Train steps: 7339

This repo contains the final saved SFT checkpoint from /posttraining/checkpoints/sft-tulu3-4n-gbs128.