karpathy/gpt2_1558M_final4_hf
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: karpathy/gpt2_1558M_final4_hf
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-10-05 04:33:19 +08:00
commit 7b8c4b827b
9 changed files with 100393 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
README.md Normal file
View File

@@ -0,0 +1,12 @@
---
library_name: transformers
tags: []
---
This is a GPT-2 model trained in llm.c for 330K steps (of 1M batch size) on FineWeb-EDU.
A lot more detailed information is here: https://github.com/karpathy/llm.c/discussions/677 .
This model has a bit of a complicated history. I wanted to train it for 400K steps, i.e. (`-x 400000`), but it became unstable later in training and exploded around step 330K. Because I was losing my computing quota shortly, I decided to just rewind back to checkpoint 300K, and then instead of going all the way to 400K I started annealing linearly down to 330K. This went without incident and produced this model.
This is the longest I've trained a GPT-2 model for, and it reaches HellaSwag of 62.7 by the end.