63 lines
2.3 KiB
Markdown
63 lines
2.3 KiB
Markdown
---
|
|
license: apache-2.0
|
|
language:
|
|
- en
|
|
library_name: transformers
|
|
pipeline_tag: text-generation
|
|
tags:
|
|
- llama
|
|
- causal-lm
|
|
- base-model
|
|
- continued-pretraining
|
|
- midtraining
|
|
---
|
|
|
|
# d24-climbmix-dolmino-midtrain-100b
|
|
|
|
A **~0.75B-parameter** (756,819,456) decoder-only LM. This is the [d24-climbmix-100b](https://huggingface.co/ftajwar/d24-climbmix-100b)
|
|
base **continue-pretrained ("midtrained") for another ~100B tokens** on a high-quality reasoning/instruction
|
|
blend with pretrain-data replay.
|
|
|
|
- **Base:** d24 pretrained from scratch on **100B tokens of ClimbMix**.
|
|
- **Midtrain:** **+100B tokens** as a *second pretraining phase* — weights-only reload of the base,
|
|
fresh optimizer, re-warmed to the original peak LR and re-decayed (Ibrahim et al. 2024; Gupta et al. 2023).
|
|
- **Data mixture (true per-batch blend):** **80% [OLMo-3 Dolmino](https://huggingface.co/datasets/allenai/dolma3_dolmino_mix-100B-1025)**
|
|
(reasoning/instruction, at its final 24-component proportions) **+ 20% ClimbMix replay** (the base's exact
|
|
pretrain distribution, to anchor against catastrophic forgetting).
|
|
|
|
This is a **base model** (no instruction/chat tuning, no chat template) intended for research and further
|
|
SFT/RL. Total training ≈ **200B tokens**.
|
|
|
|
## Architecture
|
|
| | |
|
|
|---|---|
|
|
| Layers | 24 |
|
|
| Hidden | 1536 |
|
|
| Heads | 12 (MHA) |
|
|
| FFN | 4096 (SwiGLU / silu) |
|
|
| Pos. enc. | RoPE (θ=10000) |
|
|
| Norm | RMSNorm |
|
|
| Embeddings | tied |
|
|
| Vocab | 50304 (gpt2 tokenizer) |
|
|
| Seq len | 2048 |
|
|
| Dtype | bf16 |
|
|
|
|
## Midtrain hyperparameters
|
|
Global batch 1024 · seq 2048 · **47,684 steps (~100B tokens)** · AdamW β(0.9, 0.95), wd 0.1, grad-clip 1.0 ·
|
|
peak LR **3e-4** → min **3e-5**, cosine · warmup **2000** steps. Peak LR validated stable via an LR-ablation smoke.
|
|
|
|
## Usage
|
|
```python
|
|
import torch
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
mid = "ftajwar/d24-climbmix-dolmino-midtrain-100b"
|
|
tok = AutoTokenizer.from_pretrained(mid)
|
|
model = AutoModelForCausalLM.from_pretrained(mid, torch_dtype=torch.bfloat16).cuda().eval()
|
|
ids = tok("The first three prime numbers are", return_tensors="pt").to("cuda")
|
|
print(tok.decode(model.generate(**ids, max_new_tokens=40)[0], skip_special_tokens=True))
|
|
```
|
|
|
|
## Tokenizer
|
|
gpt2 BPE (vocab padded to 50304), `--append-eod` during data prep. It is a **base** completion model —
|
|
prompt it with plain text, not chat turns.
|