---
license: mit
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- gpt2
- fineweb-edu
- pretrained
---

# GPT-2 124M FineWeb-Edu 10B

This is a GPT-2 124M **base text-completion model trained from scratch** on the FineWeb-Edu `sample-10BT` dataset. It is not instruction-tuned and is not a chatbot.

## Try it live

Run this model in a free public Gradio Space:

**[Open the live demo →](https://huggingface.co/spaces/ctxnn1/gpt2-from-scratch-demo)**

> The Space runs on CPU-basic (free) hardware and accepts a prompt with
> temperature, top-k, top-p, repetition penalty, and seed controls.

## Model description

- Architecture: GPT-2 decoder-only Transformer
- Parameters: 124,439,808 after export-vocabulary trimming
- Layers / heads / hidden size: 12 / 12 / 768
- Context length: 1,024 tokens
- Export vocabulary: 50,257 GPT-2 tokens
- Training tokens: 9,999,745,024
- Hardware: one NVIDIA H100

The native trainer padded its embedding/output matrix from 50,257 to 50,304 rows for efficient kernels. The final 47 rows were never tokenizer-addressable. This export keeps rows 0–50,256, sets `config.vocab_size=50257`, and preserves tied input/output embeddings, so generation cannot emit a padded ID.

## Final evaluation

| Metric | Value |
|---|---:|
| Training step | 19,073 |
| Train loss | 3.103327 |
| Validation loss | 3.030832 |
| Validation perplexity | 20.714451 |
| HellaSwag accuracy | 30.0339% (3,016/10,042) |

## Intended use

The model is intended for research, education, reproducibility studies, and experiments with small pretrained language models. It performs ordinary next-token completion.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "ctxnn1/gpt2-124m-fineweb-edu-10b"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
inputs = tokenizer("The future of artificial intelligence is", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=64, do_sample=True, top_k=50)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```

## Limitations, risks, and biases

- This is a 124M-parameter base model and is not competitive with modern large language models.
- It is not instruction-tuned, preference-aligned, safety-tuned, or suitable as a chatbot.
- Outputs can be inaccurate, incoherent, biased, offensive, unsafe, or memorized from pretraining data.
- HellaSwag accuracy is only modestly above the 25% random-choice baseline.
- The run did not include comprehensive safety, fairness, memorization, or downstream-task evaluation.
- Users must evaluate outputs and suitability for their own domain before deployment.

## Training and conversion provenance

- Source code: https://github.com/ctxnn/gpt-2
- W&B: https://wandb.ai/ctxnn-thapar-university/gpt2-from-scratch/runs/65e78f54c14046ef99e04e12e7b3e810
- Native checkpoint SHA-256: `e519d993d20c98c841ef061f76a1dec3e6ee24d5e55162bdea2a3e2da280fd40`
- Training Git SHA: `36bfc9edd044eb828e118d49c79532eef8440a2a`
- Cloud execution Git SHA: `9862792ab4024f9ebec758be73ebe7e75419d09b`

The native Linear weights for attention and MLP projections were transposed into Hugging Face GPT-2 `Conv1D` orientation. Positional embeddings, LayerNorm parameters, attention/MLP projections, and tied token embeddings were preserved and validated with native-versus-Hugging-Face logit and loss comparisons.