---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- llama
- causal-lm
- base-model
- continued-pretraining
- midtraining
---

# d24-climbmix-dolmino-midtrain-100b

A **~0.75B-parameter** (756,819,456) decoder-only LM. This is the [d24-climbmix-100b](https://huggingface.co/ftajwar/d24-climbmix-100b)
base **continue-pretrained ("midtrained") for another ~100B tokens** on a high-quality reasoning/instruction
blend with pretrain-data replay.

- **Base:** d24 pretrained from scratch on **100B tokens of ClimbMix**.
- **Midtrain:** **+100B tokens** as a *second pretraining phase* — weights-only reload of the base,
  fresh optimizer, re-warmed to the original peak LR and re-decayed (Ibrahim et al. 2024; Gupta et al. 2023).
- **Data mixture (true per-batch blend):** **80% [OLMo-3 Dolmino](https://huggingface.co/datasets/allenai/dolma3_dolmino_mix-100B-1025)**
  (reasoning/instruction, at its final 24-component proportions) **+ 20% ClimbMix replay** (the base's exact
  pretrain distribution, to anchor against catastrophic forgetting).

This is a **base model** (no instruction/chat tuning, no chat template) intended for research and further
SFT/RL. Total training ≈ **200B tokens**.

## Architecture
| | |
|---|---|
| Layers | 24 |
| Hidden | 1536 |
| Heads | 12 (MHA) |
| FFN | 4096 (SwiGLU / silu) |
| Pos. enc. | RoPE (θ=10000) |
| Norm | RMSNorm |
| Embeddings | tied |
| Vocab | 50304 (gpt2 tokenizer) |
| Seq len | 2048 |
| Dtype | bf16 |

## Midtrain hyperparameters
Global batch 1024 · seq 2048 · **47,684 steps (~100B tokens)** · AdamW β(0.9, 0.95), wd 0.1, grad-clip 1.0 ·
peak LR **3e-4** → min **3e-5**, cosine · warmup **2000** steps. Peak LR validated stable via an LR-ablation smoke.

## Usage
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
mid = "ftajwar/d24-climbmix-dolmino-midtrain-100b"
tok = AutoTokenizer.from_pretrained(mid)
model = AutoModelForCausalLM.from_pretrained(mid, torch_dtype=torch.bfloat16).cuda().eval()
ids = tok("The first three prime numbers are", return_tensors="pt").to("cuda")
print(tok.decode(model.generate(**ids, max_new_tokens=40)[0], skip_special_tokens=True))
```

## Tokenizer
gpt2 BPE (vocab padded to 50304), `--append-eod` during data prep. It is a **base** completion model —
prompt it with plain text, not chat turns.