41 lines
880 B
Markdown
41 lines
880 B
Markdown
|
---
|
||
|
|
language: en
|
||
|
|
tags:
|
||
|
|
- causal-lm
|
||
|
|
- gqa
|
||
|
|
- rope
|
||
|
|
- swiglu
|
||
|
|
license: apache-2.0
|
||
|
|
---
|
||
|
|
|
||
|
|
# Petal-50M
|
||
|
|
|
||
|
|
Small language model (49.7M parameters) trained from scratch.
|
||
|
|
|
||
|
|
## Architecture
|
||
|
|
| Property | Value |
|
||
|
|
|---|---|
|
||
|
|
| Layers | 14 |
|
||
|
|
| Hidden size | 512 |
|
||
|
|
| Intermediate size | 1408 |
|
||
|
|
| Attention heads | 8 (GQA kv=4) |
|
||
|
|
| Max sequence length | 1024 |
|
||
|
|
| Vocab size | 16384 |
|
||
|
|
| Tied embeddings | True |
|
||
|
|
| Total parameters | 49.691M |
|
||
|
|
|
||
|
|
## Training
|
||
|
|
- Tokens seen: 12,206,075,904
|
||
|
|
- Val loss: 1.8920
|
||
|
|
- Val PPL: 6.63
|
||
|
|
|
||
|
|
## Usage
|
||
|
|
```python
|
||
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||
|
|
tokenizer = AutoTokenizer.from_pretrained("CyanMonkey/Petal-50M")
|
||
|
|
model = AutoModelForCausalLM.from_pretrained("CyanMonkey/Petal-50M")
|
||
|
|
inputs = tokenizer("Hello", return_tensors="pt")
|
||
|
|
output = model.generate(**inputs, max_new_tokens=50)
|
||
|
|
print(tokenizer.decode(output[0], skip_special_tokens=True))
|
||
|
|
```
|