Raghav-Singhal/pretrain-normal-smollm-1p7b-100B-20n-2048sl-960gbsz
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity
1 Commit 1 Branch 0 Tags
main
Go to file
Clone
Open with VS Code Open with VSCodium Open with Intellij IDEA
Download ZIP Download TAR.GZ Download BUNDLE

README.md

library_name, pipeline_tag, tags
library_name pipeline_tag tags
transformers text-generation
llama
causal-lm
bfloat16

pretrain-normal-smollm-1p7b-100B-20n-2048sl-960gbsz

Converted Hugging Face base checkpoint from the Model Raising pretraining run.

Details

  • Architecture: LlamaForCausalLM
  • Base model size: 1.7B
  • Precision on disk: bfloat16
  • Tokenizer: HuggingFaceTB/SmolLM2-1.7B-Instruct

This repo contains the verified Hugging Face export of the final pretraining checkpoint before SFT.

Description
Model synced from source: Raghav-Singhal/pretrain-normal-smollm-1p7b-100B-20n-2048sl-960gbsz
Readme 1.3 MiB
Languages
Jinja 100%