初始化项目，由ModelHub XC社区提供模型

Model: sfanm/d24-sft-v5-simplechat Source: Original Platform
2026-07-27 05:36:10 +08:00
commit d11669d962
8 changed files with 250577 additions and 0 deletions
--- a/README.md
+++ b/README.md
@@ -0,0 +1,180 @@
+---
+license: other
+language:
+- en
+library_name: transformers
+pipeline_tag: text-generation
+base_model: ftajwar/d24-climbmix-dolmino-midtrain-100b
+datasets:
+- sfanm/d24-sft-mixture
+tags:
+- llama
+- causal-lm
+- chat
+- sft
+- nanochat
+- nemotron
+- gpt2-tokenizer
+---
+
+# d24-sft-v5-simplechat
+
+`d24-sft-v5-simplechat` is an experimental **756,819,456-parameter** English
+chat model for research on supervised fine-tuning and reinforcement learning.
+It is the canonical simple-ChatML SFT policy from the d24 v5 lineage.
+
+All 756,819,456 parameters in the published safetensors file are BF16.
+
+## Lineage
+
+1. [`ftajwar/d24-climbmix-100b`](https://huggingface.co/ftajwar/d24-climbmix-100b):
+   pretrained from scratch on about 100B tokens of ClimbMix.
+2. [`ftajwar/d24-climbmix-dolmino-midtrain-100b`](https://huggingface.co/ftajwar/d24-climbmix-dolmino-midtrain-100b):
+   continued-pretrained for another approximately 100B tokens on 80% OLMo-3
+   Dolmino plus 20% ClimbMix replay.
+3. This checkpoint: one epoch of SFT on the standard d24 conversation mixture,
+   formatted with the simple ChatML template.
+
+The model has seen approximately 200B pretraining/midtraining tokens before SFT.
+The `v5` label identifies this training lineage; it is not a parameter-count
+label.
+
+## Architecture
+
+| Field | Value |
+|---|---:|
+| Parameters | 756,819,456 |
+| Layers | 24 |
+| Hidden size | 1,536 |
+| Attention heads | 12 (MHA) |
+| FFN size | 4,096 (SwiGLU/SiLU) |
+| Position encoding | RoPE, theta 10,000 |
+| Normalization | RMSNorm |
+| Embeddings | Tied |
+| Tokenizer | GPT-2 BPE, vocabulary padded to 50,304 |
+| Maximum context | 2,048 tokens |
+| Published weight dtype | BF16 |
+
+The Hugging Face architecture is `LlamaForCausalLM`, but the model uses the
+GPT-2 tokenizer and the d24 architecture above. It is not a Llama-family
+pretrained checkpoint.
+
+## SFT data and training
+
+The source conversations are the
+[`sfanm/d24-sft-mixture`](https://huggingface.co/datasets/sfanm/d24-sft-mixture)
+research mixture: SmolTalk, MMLU auxiliary data, GSM8K, spelling tasks, and
+identity examples. The local simple-ChatML preparation produced 227,443 packed
+training sequences and 7,381 packed validation sequences at length 2,048, with
+about 476M tokens across both splits.
+
+Training used:
+
+| Hyperparameter | Value |
+|---|---:|
+| Epochs / steps | 1 / 1,777 |
+| Global / micro batch | 128 / 1 |
+| Sequence length | 2,048 |
+| Optimizer | AdamW |
+| Peak / minimum LR | 1e-4 / 1e-5 |
+| LR schedule | Cosine, 50-step warmup |
+| Adam betas / epsilon | (0.9, 0.95) / 1e-8 |
+| Weight decay | 0.1 |
+| Gradient clip | 1.0 |
+| Precision | BF16 |
+
+The final SFT validation language-model loss was **0.79675** (perplexity
+**2.2183**). This is an in-distribution SFT validation metric, not a downstream
+capability or safety evaluation.
+
+## Chat format and stopping
+
+The template is:
+
+```text
+<|im_start|>user
+...<|im_end|>
+<|im_start|>assistant
+...
+```
+
+`<|im_end|>` is a literal string represented by seven GPT-2 BPE tokens. It is
+**not** `eos_token_id=50256` and is not a registered special token. Generation
+must stop on the string `<|im_end|>`; otherwise the model can continue into
+another turn until `max_new_tokens`.
+
+## Transformers usage
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "sfanm/d24-sft-v5-simplechat"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+).eval()
+
+messages = [
+    {
+        "role": "user",
+        "content": "Natalia sold clips to 48 friends in April and half as many in May. How many total?",
+    }
+]
+prompt = tokenizer.apply_chat_template(
+    messages,
+    tokenize=False,
+    add_generation_prompt=True,
+)
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+output = model.generate(
+    **inputs,
+    max_new_tokens=512,
+    do_sample=False,
+    stop_strings=["<|im_end|>"],
+    tokenizer=tokenizer,
+)
+response = tokenizer.decode(
+    output[0, inputs.input_ids.shape[1]:],
+    skip_special_tokens=False,
+)
+print(response)
+```
+
+For vLLM, use `SamplingParams(stop=["<|im_end|>"])`.
+
+## Intended use
+
+- research on SFT, RLVR, GRPO/RLOO/maxRL, and small-model behavior;
+- a warm-start chat policy for the d24 regular-RL recipes;
+- controlled comparisons with other d24 SFT checkpoints.
+
+The accompanying `unified-training` repository contains the reproducible
+cluster setup, current RL defaults, and algorithm baselines in
+`docs/REGULAR_RL_TRAINING.md`.
+
+## Limitations and evaluation caveats
+
+- This is a small experimental research model and has not undergone production
+  safety alignment or a comprehensive capability/safety evaluation.
+- It can produce incorrect, incoherent, biased, or unsafe text. Verify outputs
+  independently and do not use it for high-stakes decisions.
+- The context window is 2,048 tokens.
+- The SFT mixture includes GSM8K training examples. Do not treat GSM8K results
+  from this checkpoint as a clean no-exposure benchmark. The separate local
+  `d24-sft-v5-nogsm8k-hf` checkpoint exists for that control but is not part of
+  this repository.
+- Chat generation is incorrect unless the caller stops on the literal
+  `<|im_end|>` string.
+- The model and SFT mixture derive from multiple upstream datasets. The Hub
+  metadata therefore uses `license: other`; review the source dataset licenses
+  and terms before redistribution or downstream use.
+
+## Reproducibility
+
+- SFT checkpoint: `sft-d24-v5-simplechat/iter_0001777`
+- HF export: `d24-sft-v5-simplechat-hf`
+- Chat template: `src/nemotron/data_prep/templates/simple_chatml.jinja`
+- Training repository: `yudasong/unified-training`