117 lines
3.5 KiB
Markdown
117 lines
3.5 KiB
Markdown
---
|
|
language: pt
|
|
license: mit
|
|
tags:
|
|
- gpt2
|
|
- portuguese
|
|
- causal-lm
|
|
- pytorch
|
|
- safetensors
|
|
pipeline_tag: text-generation
|
|
---
|
|
|
|
# gpt2-ptbr-218m
|
|
|
|
Portuguese GPT-2-like autoregressive language model trained from scratch.
|
|
|
|
## Training pipeline
|
|
|
|
The released model, `gpt2-ptbr-218m`, is the final checkpoint of a three-stage pipeline:
|
|
|
|
1. **Pretraining** on Portuguese Wikipedia.
|
|
2. **Supervised Fine-Tuning** on Alpaca PT-BR.
|
|
3. **Supervised Fine-Tuning** on Canarim-Instruct-PTBR.
|
|
|
|
The released checkpoint corresponds to the final instruction-tuned model.
|
|
|
|
## Model description
|
|
|
|
- **Architecture:** GPT-2 (decoder-only transformer)
|
|
- **Parameters:** 218,040,320 trainable unique parameters (weight-tying between token embedding and output projection; zero biases added for Hugging Face compatibility)
|
|
- **Layers:** 16
|
|
- **Attention heads:** 16
|
|
- **Embedding dimension:** 1024
|
|
- **Vocabulary:** 16,000 tokens (SentencePiece BPE)
|
|
- **Sequence length:** 256 tokens
|
|
- **Activation:** GELU
|
|
- **Dropout:** 0.1
|
|
|
|
> **Note on parameter count:** During export, the weight-tying between the token embedding and the language model head is preserved, and zero-initialized bias tensors are added for compatibility with Hugging Face's `GPT2LMHeadModel`. These biases are not part of the original trained model and do not affect behavior.
|
|
|
|
## Datasets
|
|
|
|
- **Portuguese Wikipedia corpus** — used for autoregressive pretraining.
|
|
- **Alpaca PT-BR** — Portuguese instruction-following dataset derived from Stanford Alpaca (Taori et al., 2023). Used for the first SFT stage.
|
|
- **Canarim-Instruct-PTBR** — Portuguese instruction-following dataset by Maicon Domingues (Domingues, 2023). Used for the second SFT stage.
|
|
|
|
## Tokenizer
|
|
|
|
- **Type:** SentencePiece BPE
|
|
- **Vocabulary:** 16,000 tokens
|
|
- **Special tokens:** `<unk>` = 0, `<bos>` = 1, `<eos>` = 2, `<pad>` = 3
|
|
- **Pre-tokenizer:** Metaspace (SentencePiece native)
|
|
- **Model max length:** 256 tokens
|
|
|
|
The tokenizer was trained from scratch on the Portuguese Wikipedia corpus.
|
|
|
|
## Training details
|
|
|
|
- **Type:** sft_response_only
|
|
- **Best validation loss:** 2.1538288593292236
|
|
- **Training steps:** 1050
|
|
|
|
## Usage
|
|
|
|
```python
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
model_id = "augustoafleal/gpt2-ptbr-218m"
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
|
model = AutoModelForCausalLM.from_pretrained(model_id)
|
|
|
|
prompt = "A inteligência artificial é"
|
|
inputs = tokenizer(prompt, return_tensors="pt")
|
|
|
|
output = model.generate(
|
|
**inputs,
|
|
max_new_tokens=100,
|
|
do_sample=True,
|
|
temperature=0.7,
|
|
top_k=40,
|
|
pad_token_id=tokenizer.pad_token_id,
|
|
eos_token_id=tokenizer.eos_token_id,
|
|
)
|
|
|
|
print(tokenizer.decode(output[0], skip_special_tokens=True))
|
|
```
|
|
|
|
## Limitations
|
|
|
|
- The model may generate factual errors.
|
|
- The model may repeat phrases.
|
|
- The model may fail to follow instructions exactly.
|
|
- The context length is limited to 256 tokens.
|
|
- The model was trained on limited data compared to modern LLMs.
|
|
- This model is not suitable for high-stakes use without human validation.
|
|
|
|
## Dataset citations
|
|
|
|
```
|
|
Stanford Alpaca: Taori et al., 2023. https://github.com/tatsu-lab/stanford_alpaca
|
|
Canarim-Instruct-PTBR: Domingues, 2023. https://huggingface.co/datasets/dominguesm/Canarim-Instruct-PTBR
|
|
```
|
|
|
|
## Citation
|
|
|
|
If you use this model, please cite:
|
|
|
|
```bibtex
|
|
@misc{gpt2ptbr218m,
|
|
title = {gpt2-ptbr-218m: A Portuguese GPT-2-like Autoregressive Language Model},
|
|
author = {Augusto Antônio Fontanive Leal},
|
|
year = {2026},
|
|
howpublished = {\url{https://huggingface.co/augustoafleal/gpt2-ptbr-218m}}
|
|
}
|
|
```
|