初始化项目，由ModelHub XC社区提供模型

Model: sfanm/d24-sft-v5-simplechat
Source: Original Platform

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,180 @@
+---
+license: other
+language:
+- en
+library_name: transformers
+pipeline_tag: text-generation
+base_model: ftajwar/d24-climbmix-dolmino-midtrain-100b
+datasets:
+- sfanm/d24-sft-mixture
+tags:
+- llama
+- causal-lm
+- chat
+- sft
+- nanochat
+- nemotron
+- gpt2-tokenizer
+---
+
+# d24-sft-v5-simplechat
+
+`d24-sft-v5-simplechat` is an experimental **756,819,456-parameter** English
+chat model for research on supervised fine-tuning and reinforcement learning.
+It is the canonical simple-ChatML SFT policy from the d24 v5 lineage.
+
+All 756,819,456 parameters in the published safetensors file are BF16.
+
+## Lineage
+
+1. [`ftajwar/d24-climbmix-100b`](https://huggingface.co/ftajwar/d24-climbmix-100b):
+   pretrained from scratch on about 100B tokens of ClimbMix.
+2. [`ftajwar/d24-climbmix-dolmino-midtrain-100b`](https://huggingface.co/ftajwar/d24-climbmix-dolmino-midtrain-100b):
+   continued-pretrained for another approximately 100B tokens on 80% OLMo-3
+   Dolmino plus 20% ClimbMix replay.
+3. This checkpoint: one epoch of SFT on the standard d24 conversation mixture,
+   formatted with the simple ChatML template.
+
+The model has seen approximately 200B pretraining/midtraining tokens before SFT.
+The `v5` label identifies this training lineage; it is not a parameter-count
+label.
+
+## Architecture
+
+| Field | Value |
+|---|---:|
+| Parameters | 756,819,456 |
+| Layers | 24 |
+| Hidden size | 1,536 |
+| Attention heads | 12 (MHA) |
+| FFN size | 4,096 (SwiGLU/SiLU) |
+| Position encoding | RoPE, theta 10,000 |
+| Normalization | RMSNorm |
+| Embeddings | Tied |
+| Tokenizer | GPT-2 BPE, vocabulary padded to 50,304 |
+| Maximum context | 2,048 tokens |
+| Published weight dtype | BF16 |
+
+The Hugging Face architecture is `LlamaForCausalLM`, but the model uses the
+GPT-2 tokenizer and the d24 architecture above. It is not a Llama-family
+pretrained checkpoint.
+
+## SFT data and training
+
+The source conversations are the
+[`sfanm/d24-sft-mixture`](https://huggingface.co/datasets/sfanm/d24-sft-mixture)
+research mixture: SmolTalk, MMLU auxiliary data, GSM8K, spelling tasks, and
+identity examples. The local simple-ChatML preparation produced 227,443 packed
+training sequences and 7,381 packed validation sequences at length 2,048, with
+about 476M tokens across both splits.
+
+Training used:
+
+| Hyperparameter | Value |
+|---|---:|
+| Epochs / steps | 1 / 1,777 |
+| Global / micro batch | 128 / 1 |
+| Sequence length | 2,048 |
+| Optimizer | AdamW |
+| Peak / minimum LR | 1e-4 / 1e-5 |
+| LR schedule | Cosine, 50-step warmup |
+| Adam betas / epsilon | (0.9, 0.95) / 1e-8 |
+| Weight decay | 0.1 |
+| Gradient clip | 1.0 |
+| Precision | BF16 |
+
+The final SFT validation language-model loss was **0.79675** (perplexity
+**2.2183**). This is an in-distribution SFT validation metric, not a downstream
+capability or safety evaluation.
+
+## Chat format and stopping
+
+The template is:
+
+```text
+<|im_start|>user
+...<|im_end|>
+<|im_start|>assistant
+...
+```
+
+`<|im_end|>` is a literal string represented by seven GPT-2 BPE tokens. It is
+**not** `eos_token_id=50256` and is not a registered special token. Generation
+must stop on the string `<|im_end|>`; otherwise the model can continue into
+another turn until `max_new_tokens`.
+
+## Transformers usage
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "sfanm/d24-sft-v5-simplechat"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+).eval()
+
+messages = [
+    {
+        "role": "user",
+        "content": "Natalia sold clips to 48 friends in April and half as many in May. How many total?",
+    }
+]
+prompt = tokenizer.apply_chat_template(
+    messages,
+    tokenize=False,
+    add_generation_prompt=True,
+)
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+output = model.generate(
+    **inputs,
+    max_new_tokens=512,
+    do_sample=False,
+    stop_strings=["<|im_end|>"],
+    tokenizer=tokenizer,
+)
+response = tokenizer.decode(
+    output[0, inputs.input_ids.shape[1]:],
+    skip_special_tokens=False,
+)
+print(response)
+```
+
+For vLLM, use `SamplingParams(stop=["<|im_end|>"])`.
+
+## Intended use
+
+- research on SFT, RLVR, GRPO/RLOO/maxRL, and small-model behavior;
+- a warm-start chat policy for the d24 regular-RL recipes;
+- controlled comparisons with other d24 SFT checkpoints.
+
+The accompanying `unified-training` repository contains the reproducible
+cluster setup, current RL defaults, and algorithm baselines in
+`docs/REGULAR_RL_TRAINING.md`.
+
+## Limitations and evaluation caveats
+
+- This is a small experimental research model and has not undergone production
+  safety alignment or a comprehensive capability/safety evaluation.
+- It can produce incorrect, incoherent, biased, or unsafe text. Verify outputs
+  independently and do not use it for high-stakes decisions.
+- The context window is 2,048 tokens.
+- The SFT mixture includes GSM8K training examples. Do not treat GSM8K results
+  from this checkpoint as a clean no-exposure benchmark. The separate local
+  `d24-sft-v5-nogsm8k-hf` checkpoint exists for that control but is not part of
+  this repository.
+- Chat generation is incorrect unless the caller stops on the literal
+  `<|im_end|>` string.
+- The model and SFT mixture derive from multiple upstream datasets. The Hub
+  metadata therefore uses `license: other`; review the source dataset licenses
+  and terms before redistribution or downstream use.
+
+## Reproducibility
+
+- SFT checkpoint: `sft-d24-v5-simplechat/iter_0001777`
+- HF export: `d24-sft-v5-simplechat-hf`
+- Chat template: `src/nemotron/data_prep/templates/simple_chatml.jinja`
+- Training repository: `yudasong/unified-training`

chat_template.jinja

@@ -0,0 +1 @@
+{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{'<|im_start|>assistant\n'}}{% endif %}

config.json

@@ -0,0 +1,29 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 50256,
+  "dtype": "float32",
+  "eos_token_id": 50256,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 1536,
+  "initializer_range": 0.02,
+  "intermediate_size": 4096,
+  "max_position_embeddings": 2048,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 24,
+  "num_key_value_heads": 12,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": true,
+  "transformers_version": "4.57.1",
+  "use_cache": true,
+  "vocab_size": 50304
+}

generation_config.json

@@ -0,0 +1,9 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 50256,
+  "eos_token_id": 50256,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "transformers_version": "5.3.0",
+  "use_cache": true
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d847e1d6680e7a63510f70bcfcb6d14b6dd35f3bfdd4c8e524ac39af62d0d6fd
+size 1513663904

tokenizer_config.json

@@ -0,0 +1,14 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "cls_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "is_local": false,
+  "model_max_length": 2048,
+  "pad_token": null,
+  "sep_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}