24 lines
555 B
Markdown
24 lines
555 B
Markdown
---
|
|
tags:
|
|
- causal-lm
|
|
- text-generation
|
|
library_name: transformers
|
|
---
|
|
|
|
# nikitastheo/phonebabylm-phone-deu-eng-interleaved
|
|
|
|
Trained with train_clm.py, a Hugging Face Accelerate causal-LM training script (no `Trainer`).
|
|
|
|
## Training details
|
|
|
|
- **Base config**: `gpt_base_config.json`
|
|
- **Tokenizer**: `nikitastheo/phonelm-phone-deu-eng-tokenizer`
|
|
- **Max steps**: 40000
|
|
- **Learning rate**: 0.0005
|
|
- **LR scheduler**: linear
|
|
- **Warmup steps**: 4000
|
|
- **Batch size (per device)**: 32
|
|
- **Gradient accumulation steps**: 1
|
|
- **Total train batch size**: 32
|
|
|