d24-climbmix-dolmino-midtrain-100b/README.md

license, language, library_name, pipeline_tag, tags

license	language	library_name	pipeline_tag	tags
apache-2.0	en	transformers	text-generation	llama causal-lm base-model continued-pretraining midtraining

d24-climbmix-dolmino-midtrain-100b

A ~0.75B-parameter (756,819,456) decoder-only LM. This is the d24-climbmix-100b base continue-pretrained ("midtrained") for another ~100B tokens on a high-quality reasoning/instruction blend with pretrain-data replay.

• Base: d24 pretrained from scratch on 100B tokens of ClimbMix.
• Midtrain: +100B tokens as a second pretraining phase — weights-only reload of the base, fresh optimizer, re-warmed to the original peak LR and re-decayed (Ibrahim et al. 2024; Gupta et al. 2023).
• Data mixture (true per-batch blend): 80% OLMo-3 Dolmino (reasoning/instruction, at its final 24-component proportions) + 20% ClimbMix replay (the base's exact pretrain distribution, to anchor against catastrophic forgetting).

This is a base model (no instruction/chat tuning, no chat template) intended for research and further SFT/RL. Total training ≈ 200B tokens.

Architecture

Layers	24
Hidden	1536
Heads	12 (MHA)
FFN	4096 (SwiGLU / silu)
Pos. enc.	RoPE (θ=10000)
Norm	RMSNorm
Embeddings	tied
Vocab	50304 (gpt2 tokenizer)
Seq len	2048
Dtype	bf16

Midtrain hyperparameters

Global batch 1024 · seq 2048 · 47,684 steps (~100B tokens) · AdamW β(0.9, 0.95), wd 0.1, grad-clip 1.0 · peak LR 3e-4 → min 3e-5, cosine · warmup 2000 steps. Peak LR validated stable via an LR-ablation smoke.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
mid = "ftajwar/d24-climbmix-dolmino-midtrain-100b"
tok = AutoTokenizer.from_pretrained(mid)
model = AutoModelForCausalLM.from_pretrained(mid, torch_dtype=torch.bfloat16).cuda().eval()
ids = tok("The first three prime numbers are", return_tensors="pt").to("cuda")
print(tok.decode(model.generate(**ids, max_new_tokens=40)[0], skip_special_tokens=True))

Tokenizer

gpt2 BPE (vocab padded to 50304), --append-eod during data prep. It is a base completion model — prompt it with plain text, not chat turns.