yuchenj/gpt2_124M_100B_FinewebEdu_hf
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity
1 Commit 1 Branch 0 Tags
bb5bdb1c6f2b191775ad1072869d3a66e7a01237
Go to file
Clone
Open with VS Code Open with VSCodium Open with Intellij IDEA
Download ZIP Download TAR.GZ Download BUNDLE

README.md

library_name, datasets
library_name datasets
transformers
HuggingFaceFW/fineweb-edu

This is a GPT-2 (124M) model trained in llm.c for 100B tokens with cosine LR on Fineweb-Edu.

A lot more detailed info and observations are here: https://x.com/Yuchenj_UW/status/1814703583453192272

Description
Model synced from source: yuchenj/gpt2_124M_100B_FinewebEdu_hf
Readme 620 KiB
Languages
Text 100%