初始化项目，由ModelHub XC社区提供模型

Model: ftajwar/d24-climbmix-dolmino-midtrain-100b
Source: Original Platform

README.md

@@ -0,0 +1,62 @@
+---
+license: apache-2.0
+language:
+- en
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- llama
+- causal-lm
+- base-model
+- continued-pretraining
+- midtraining
+---
+
+# d24-climbmix-dolmino-midtrain-100b
+
+A **~0.75B-parameter** (756,819,456) decoder-only LM. This is the [d24-climbmix-100b](https://huggingface.co/ftajwar/d24-climbmix-100b)
+base **continue-pretrained ("midtrained") for another ~100B tokens** on a high-quality reasoning/instruction
+blend with pretrain-data replay.
+
+- **Base:** d24 pretrained from scratch on **100B tokens of ClimbMix**.
+- **Midtrain:** **+100B tokens** as a *second pretraining phase* — weights-only reload of the base,
+  fresh optimizer, re-warmed to the original peak LR and re-decayed (Ibrahim et al. 2024; Gupta et al. 2023).
+- **Data mixture (true per-batch blend):** **80% [OLMo-3 Dolmino](https://huggingface.co/datasets/allenai/dolma3_dolmino_mix-100B-1025)**
+  (reasoning/instruction, at its final 24-component proportions) **+ 20% ClimbMix replay** (the base's exact
+  pretrain distribution, to anchor against catastrophic forgetting).
+
+This is a **base model** (no instruction/chat tuning, no chat template) intended for research and further
+SFT/RL. Total training ≈ **200B tokens**.
+
+## Architecture
+| | |
+|---|---|
+| Layers | 24 |
+| Hidden | 1536 |
+| Heads | 12 (MHA) |
+| FFN | 4096 (SwiGLU / silu) |
+| Pos. enc. | RoPE (θ=10000) |
+| Norm | RMSNorm |
+| Embeddings | tied |
+| Vocab | 50304 (gpt2 tokenizer) |
+| Seq len | 2048 |
+| Dtype | bf16 |
+
+## Midtrain hyperparameters
+Global batch 1024 · seq 2048 · **47,684 steps (~100B tokens)** · AdamW β(0.9, 0.95), wd 0.1, grad-clip 1.0 ·
+peak LR **3e-4** → min **3e-5**, cosine · warmup **2000** steps. Peak LR validated stable via an LR-ablation smoke.
+
+## Usage
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+mid = "ftajwar/d24-climbmix-dolmino-midtrain-100b"
+tok = AutoTokenizer.from_pretrained(mid)
+model = AutoModelForCausalLM.from_pretrained(mid, torch_dtype=torch.bfloat16).cuda().eval()
+ids = tok("The first three prime numbers are", return_tensors="pt").to("cuda")
+print(tok.decode(model.generate(**ids, max_new_tokens=40)[0], skip_special_tokens=True))
+```
+
+## Tokenizer
+gpt2 BPE (vocab padded to 50304), `--append-eod` during data prep. It is a **base** completion model —
+prompt it with plain text, not chat turns.