初始化项目，由ModelHub XC社区提供模型

Model: ftajwar/d24-climbmix-dolmino-midtrain-100b
Source: Original Platform

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,62 @@
+---
+license: apache-2.0
+language:
+- en
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- llama
+- causal-lm
+- base-model
+- continued-pretraining
+- midtraining
+---
+
+# d24-climbmix-dolmino-midtrain-100b
+
+A **~0.75B-parameter** (756,819,456) decoder-only LM. This is the [d24-climbmix-100b](https://huggingface.co/ftajwar/d24-climbmix-100b)
+base **continue-pretrained ("midtrained") for another ~100B tokens** on a high-quality reasoning/instruction
+blend with pretrain-data replay.
+
+- **Base:** d24 pretrained from scratch on **100B tokens of ClimbMix**.
+- **Midtrain:** **+100B tokens** as a *second pretraining phase* — weights-only reload of the base,
+  fresh optimizer, re-warmed to the original peak LR and re-decayed (Ibrahim et al. 2024; Gupta et al. 2023).
+- **Data mixture (true per-batch blend):** **80% [OLMo-3 Dolmino](https://huggingface.co/datasets/allenai/dolma3_dolmino_mix-100B-1025)**
+  (reasoning/instruction, at its final 24-component proportions) **+ 20% ClimbMix replay** (the base's exact
+  pretrain distribution, to anchor against catastrophic forgetting).
+
+This is a **base model** (no instruction/chat tuning, no chat template) intended for research and further
+SFT/RL. Total training ≈ **200B tokens**.
+
+## Architecture
+| | |
+|---|---|
+| Layers | 24 |
+| Hidden | 1536 |
+| Heads | 12 (MHA) |
+| FFN | 4096 (SwiGLU / silu) |
+| Pos. enc. | RoPE (θ=10000) |
+| Norm | RMSNorm |
+| Embeddings | tied |
+| Vocab | 50304 (gpt2 tokenizer) |
+| Seq len | 2048 |
+| Dtype | bf16 |
+
+## Midtrain hyperparameters
+Global batch 1024 · seq 2048 · **47,684 steps (~100B tokens)** · AdamW β(0.9, 0.95), wd 0.1, grad-clip 1.0 ·
+peak LR **3e-4** → min **3e-5**, cosine · warmup **2000** steps. Peak LR validated stable via an LR-ablation smoke.
+
+## Usage
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+mid = "ftajwar/d24-climbmix-dolmino-midtrain-100b"
+tok = AutoTokenizer.from_pretrained(mid)
+model = AutoModelForCausalLM.from_pretrained(mid, torch_dtype=torch.bfloat16).cuda().eval()
+ids = tok("The first three prime numbers are", return_tensors="pt").to("cuda")
+print(tok.decode(model.generate(**ids, max_new_tokens=40)[0], skip_special_tokens=True))
+```
+
+## Tokenizer
+gpt2 BPE (vocab padded to 50304), `--append-eod` during data prep. It is a **base** completion model —
+prompt it with plain text, not chat turns.

config.json

@@ -0,0 +1,32 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 50256,
+  "dtype": "float32",
+  "eos_token_id": 50256,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 1536,
+  "initializer_range": 0.02,
+  "intermediate_size": 4096,
+  "max_position_embeddings": 2048,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 24,
+  "num_key_value_heads": 12,
+  "pad_token_id": null,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_parameters": {
+    "rope_theta": 10000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.3.0",
+  "use_cache": true,
+  "vocab_size": 50304
+}

generation_config.json

@@ -0,0 +1,9 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 50256,
+  "eos_token_id": 50256,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "transformers_version": "5.3.0",
+  "use_cache": true
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f5d76fd899ebd510068ecdcdb3f7d74d73a8dc924dd568120411a62f1de7bbf6
+size 1513663904

tokenizer_config.json

@@ -0,0 +1,14 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "cls_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "is_local": true,
+  "model_max_length": 1024,
+  "pad_token": null,
+  "sep_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}