545953eeab
Model: nikitastheo/phonebabylm-phone-deu-eng-sequential_interleaved Source: Original Platform
601 B
601 B
tags, library_name
| tags | library_name | ||
|---|---|---|---|
|
transformers |
nikitastheo/phonebabylm-phone-deu-eng-sequential_interleaved
Trained with train_clm.py, a Hugging Face Accelerate causal-LM training script (no Trainer).
Training details
- Base config:
gpt_base_config.json - Tokenizer:
nikitastheo/phonelm-phone-deu-eng-tokenizer - Max steps: 30000
- Learning rate: 0.0001
- LR scheduler: linear
- Warmup steps: 3000
- Batch size (per device): 32
- Gradient accumulation steps: 1
- Total train batch size: 32
- Language switch steps: 10000