Petal-50M/README.md

language, tags, license

language	tags	license
en	causal-lm gqa rope swiglu	apache-2.0

Petal-50M

Small language model (49.7M parameters) trained from scratch.

Architecture

Property	Value
Layers	14
Hidden size	512
Intermediate size	1408
Attention heads	8 (GQA kv=4)
Max sequence length	1024
Vocab size	16384
Tied embeddings	True
Total parameters	49.691M

Training

• Tokens seen: 12,206,075,904
• Val loss: 1.8920
• Val PPL: 6.63

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("CyanMonkey/Petal-50M")
model = AutoModelForCausalLM.from_pretrained("CyanMonkey/Petal-50M")
inputs = tokenizer("Hello", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(output[0], skip_special_tokens=True))