---
language: pt
license: mit
tags:
- gpt2
- portuguese
- causal-lm
- pytorch
- safetensors
pipeline_tag: text-generation
---

# gpt2-ptbr-218m

Portuguese GPT-2-like autoregressive language model trained from scratch.

## Training pipeline

The released model, `gpt2-ptbr-218m`, is the final checkpoint of a three-stage pipeline:

1. **Pretraining** on Portuguese Wikipedia.
2. **Supervised Fine-Tuning** on Alpaca PT-BR.
3. **Supervised Fine-Tuning** on Canarim-Instruct-PTBR.

The released checkpoint corresponds to the final instruction-tuned model.

## Model description

- **Architecture:** GPT-2 (decoder-only transformer)
- **Parameters:** 218,040,320 trainable unique parameters (weight-tying between token embedding and output projection; zero biases added for Hugging Face compatibility)
- **Layers:** 16
- **Attention heads:** 16
- **Embedding dimension:** 1024
- **Vocabulary:** 16,000 tokens (SentencePiece BPE)
- **Sequence length:** 256 tokens
- **Activation:** GELU
- **Dropout:** 0.1

> **Note on parameter count:** During export, the weight-tying between the token embedding and the language model head is preserved, and zero-initialized bias tensors are added for compatibility with Hugging Face's `GPT2LMHeadModel`. These biases are not part of the original trained model and do not affect behavior.

## Datasets

- **Portuguese Wikipedia corpus** — used for autoregressive pretraining.
- **Alpaca PT-BR** — Portuguese instruction-following dataset derived from Stanford Alpaca (Taori et al., 2023). Used for the first SFT stage.
- **Canarim-Instruct-PTBR** — Portuguese instruction-following dataset by Maicon Domingues (Domingues, 2023). Used for the second SFT stage.

## Tokenizer

- **Type:** SentencePiece BPE
- **Vocabulary:** 16,000 tokens
- **Special tokens:** `<unk>` = 0, `<bos>` = 1, `<eos>` = 2, `<pad>` = 3
- **Pre-tokenizer:** Metaspace (SentencePiece native)
- **Model max length:** 256 tokens

The tokenizer was trained from scratch on the Portuguese Wikipedia corpus.

## Training details

- **Type:** sft_response_only
- **Best validation loss:** 2.1538288593292236
- **Training steps:** 1050

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "augustoafleal/gpt2-ptbr-218m"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

prompt = "A inteligência artificial é"
inputs = tokenizer(prompt, return_tensors="pt")

output = model.generate(
    **inputs,
    max_new_tokens=100,
    do_sample=True,
    temperature=0.7,
    top_k=40,
    pad_token_id=tokenizer.pad_token_id,
    eos_token_id=tokenizer.eos_token_id,
)

print(tokenizer.decode(output[0], skip_special_tokens=True))
```

## Limitations

- The model may generate factual errors.
- The model may repeat phrases.
- The model may fail to follow instructions exactly.
- The context length is limited to 256 tokens.
- The model was trained on limited data compared to modern LLMs.
- This model is not suitable for high-stakes use without human validation.

## Dataset citations

```
Stanford Alpaca: Taori et al., 2023. https://github.com/tatsu-lab/stanford_alpaca
Canarim-Instruct-PTBR: Domingues, 2023. https://huggingface.co/datasets/dominguesm/Canarim-Instruct-PTBR
```

## Citation

If you use this model, please cite:

```bibtex
@misc{gpt2ptbr218m,
  title        = {gpt2-ptbr-218m: A Portuguese GPT-2-like Autoregressive Language Model},
  author       = {Augusto Antônio Fontanive Leal},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/augustoafleal/gpt2-ptbr-218m}}
}
```