50 lines
1.5 KiB
Markdown
50 lines
1.5 KiB
Markdown
---
|
|
library_name: transformers
|
|
tags:
|
|
- convergent-evolution
|
|
- fourier-features
|
|
- number-embeddings
|
|
license: mit
|
|
datasets:
|
|
- deqing/addition_dataset
|
|
---
|
|
|
|
# convergent-llama-300M-muon-addition
|
|
|
|
A 300M-parameter language model trained from scratch on **[deqing/addition_dataset](https://huggingface.co/datasets/deqing/addition_dataset)** as part of the *Convergent Evolution* project, which investigates how Fourier features emerge in LLM number embeddings.
|
|
|
|
## Model details
|
|
|
|
| | |
|
|
|---|---|
|
|
| **Architecture** | LLaMA-style Transformer (12 layers, 1024 hidden, 16 heads, GQA) |
|
|
| **Parameters** | ~300M |
|
|
| **Optimizer** | Muon (for 2D weights) + AdamW (for embeddings/bias/norm) |
|
|
| **Data perturbation** | addition-only data (e.g., '123 + 456 = 579') |
|
|
| **Training data** | [deqing/addition_dataset](https://huggingface.co/datasets/deqing/addition_dataset) |
|
|
| **Context length** | 1024 |
|
|
| **Tokenizer** | Llama 3 (128K vocab) |
|
|
| **Batch size** | 512 sequences |
|
|
|
|
## Usage
|
|
|
|
```python
|
|
from transformers import AutoModelForCausalLM
|
|
|
|
# Load final checkpoint
|
|
model = AutoModelForCausalLM.from_pretrained("deqing/convergent-llama-300M-muon-addition")
|
|
```
|
|
|
|
## Training dynamics
|
|
|
|
Intermediate checkpoints are saved as branches: `tokens-200M`, `tokens-400M`, ..., `tokens-5.0B`.
|
|
|
|
```python
|
|
# Load intermediate checkpoint (e.g., at 1B tokens)
|
|
model = AutoModelForCausalLM.from_pretrained("deqing/convergent-llama-300M-muon-addition", revision="tokens-1B")
|
|
```
|
|
|
|
## Citation
|
|
|
|
Paper forthcoming.
|