初始化项目，由ModelHub XC社区提供模型
Model: sfanm/d24-sft-v5-simplechat Source: Original Platform
This commit is contained in:
ModelHub XC
35
.gitattributes
vendored
Normal file
35
.gitattributes
vendored
Normal file
@@ -0,0 +1,35 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
180
README.md
Normal file
180
README.md
Normal file
@@ -0,0 +1,180 @@
|
||||
---
|
||||
license: other
|
||||
language:
|
||||
- en
|
||||
library_name: transformers
|
||||
pipeline_tag: text-generation
|
||||
base_model: ftajwar/d24-climbmix-dolmino-midtrain-100b
|
||||
datasets:
|
||||
- sfanm/d24-sft-mixture
|
||||
tags:
|
||||
- llama
|
||||
- causal-lm
|
||||
- chat
|
||||
- sft
|
||||
- nanochat
|
||||
- nemotron
|
||||
- gpt2-tokenizer
|
||||
---
|
||||
|
||||
# d24-sft-v5-simplechat
|
||||
|
||||
`d24-sft-v5-simplechat` is an experimental **756,819,456-parameter** English
|
||||
chat model for research on supervised fine-tuning and reinforcement learning.
|
||||
It is the canonical simple-ChatML SFT policy from the d24 v5 lineage.
|
||||
|
||||
All 756,819,456 parameters in the published safetensors file are BF16.
|
||||
|
||||
## Lineage
|
||||
|
||||
1. [`ftajwar/d24-climbmix-100b`](https://huggingface.co/ftajwar/d24-climbmix-100b):
|
||||
pretrained from scratch on about 100B tokens of ClimbMix.
|
||||
2. [`ftajwar/d24-climbmix-dolmino-midtrain-100b`](https://huggingface.co/ftajwar/d24-climbmix-dolmino-midtrain-100b):
|
||||
continued-pretrained for another approximately 100B tokens on 80% OLMo-3
|
||||
Dolmino plus 20% ClimbMix replay.
|
||||
3. This checkpoint: one epoch of SFT on the standard d24 conversation mixture,
|
||||
formatted with the simple ChatML template.
|
||||
|
||||
The model has seen approximately 200B pretraining/midtraining tokens before SFT.
|
||||
The `v5` label identifies this training lineage; it is not a parameter-count
|
||||
label.
|
||||
|
||||
## Architecture
|
||||
|
||||
| Field | Value |
|
||||
|---|---:|
|
||||
| Parameters | 756,819,456 |
|
||||
| Layers | 24 |
|
||||
| Hidden size | 1,536 |
|
||||
| Attention heads | 12 (MHA) |
|
||||
| FFN size | 4,096 (SwiGLU/SiLU) |
|
||||
| Position encoding | RoPE, theta 10,000 |
|
||||
| Normalization | RMSNorm |
|
||||
| Embeddings | Tied |
|
||||
| Tokenizer | GPT-2 BPE, vocabulary padded to 50,304 |
|
||||
| Maximum context | 2,048 tokens |
|
||||
| Published weight dtype | BF16 |
|
||||
|
||||
The Hugging Face architecture is `LlamaForCausalLM`, but the model uses the
|
||||
GPT-2 tokenizer and the d24 architecture above. It is not a Llama-family
|
||||
pretrained checkpoint.
|
||||
|
||||
## SFT data and training
|
||||
|
||||
The source conversations are the
|
||||
[`sfanm/d24-sft-mixture`](https://huggingface.co/datasets/sfanm/d24-sft-mixture)
|
||||
research mixture: SmolTalk, MMLU auxiliary data, GSM8K, spelling tasks, and
|
||||
identity examples. The local simple-ChatML preparation produced 227,443 packed
|
||||
training sequences and 7,381 packed validation sequences at length 2,048, with
|
||||
about 476M tokens across both splits.
|
||||
|
||||
Training used:
|
||||
|
||||
| Hyperparameter | Value |
|
||||
|---|---:|
|
||||
| Epochs / steps | 1 / 1,777 |
|
||||
| Global / micro batch | 128 / 1 |
|
||||
| Sequence length | 2,048 |
|
||||
| Optimizer | AdamW |
|
||||
| Peak / minimum LR | 1e-4 / 1e-5 |
|
||||
| LR schedule | Cosine, 50-step warmup |
|
||||
| Adam betas / epsilon | (0.9, 0.95) / 1e-8 |
|
||||
| Weight decay | 0.1 |
|
||||
| Gradient clip | 1.0 |
|
||||
| Precision | BF16 |
|
||||
|
||||
The final SFT validation language-model loss was **0.79675** (perplexity
|
||||
**2.2183**). This is an in-distribution SFT validation metric, not a downstream
|
||||
capability or safety evaluation.
|
||||
|
||||
## Chat format and stopping
|
||||
|
||||
The template is:
|
||||
|
||||
```text
|
||||
<|im_start|>user
|
||||
...<|im_end|>
|
||||
<|im_start|>assistant
|
||||
...
|
||||
```
|
||||
|
||||
`<|im_end|>` is a literal string represented by seven GPT-2 BPE tokens. It is
|
||||
**not** `eos_token_id=50256` and is not a registered special token. Generation
|
||||
must stop on the string `<|im_end|>`; otherwise the model can continue into
|
||||
another turn until `max_new_tokens`.
|
||||
|
||||
## Transformers usage
|
||||
|
||||
```python
|
||||
import torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
model_id = "sfanm/d24-sft-v5-simplechat"
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
model_id,
|
||||
torch_dtype=torch.bfloat16,
|
||||
device_map="auto",
|
||||
).eval()
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Natalia sold clips to 48 friends in April and half as many in May. How many total?",
|
||||
}
|
||||
]
|
||||
prompt = tokenizer.apply_chat_template(
|
||||
messages,
|
||||
tokenize=False,
|
||||
add_generation_prompt=True,
|
||||
)
|
||||
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
||||
output = model.generate(
|
||||
**inputs,
|
||||
max_new_tokens=512,
|
||||
do_sample=False,
|
||||
stop_strings=["<|im_end|>"],
|
||||
tokenizer=tokenizer,
|
||||
)
|
||||
response = tokenizer.decode(
|
||||
output[0, inputs.input_ids.shape[1]:],
|
||||
skip_special_tokens=False,
|
||||
)
|
||||
print(response)
|
||||
```
|
||||
|
||||
For vLLM, use `SamplingParams(stop=["<|im_end|>"])`.
|
||||
|
||||
## Intended use
|
||||
|
||||
- research on SFT, RLVR, GRPO/RLOO/maxRL, and small-model behavior;
|
||||
- a warm-start chat policy for the d24 regular-RL recipes;
|
||||
- controlled comparisons with other d24 SFT checkpoints.
|
||||
|
||||
The accompanying `unified-training` repository contains the reproducible
|
||||
cluster setup, current RL defaults, and algorithm baselines in
|
||||
`docs/REGULAR_RL_TRAINING.md`.
|
||||
|
||||
## Limitations and evaluation caveats
|
||||
|
||||
- This is a small experimental research model and has not undergone production
|
||||
safety alignment or a comprehensive capability/safety evaluation.
|
||||
- It can produce incorrect, incoherent, biased, or unsafe text. Verify outputs
|
||||
independently and do not use it for high-stakes decisions.
|
||||
- The context window is 2,048 tokens.
|
||||
- The SFT mixture includes GSM8K training examples. Do not treat GSM8K results
|
||||
from this checkpoint as a clean no-exposure benchmark. The separate local
|
||||
`d24-sft-v5-nogsm8k-hf` checkpoint exists for that control but is not part of
|
||||
this repository.
|
||||
- Chat generation is incorrect unless the caller stops on the literal
|
||||
`<|im_end|>` string.
|
||||
- The model and SFT mixture derive from multiple upstream datasets. The Hub
|
||||
metadata therefore uses `license: other`; review the source dataset licenses
|
||||
and terms before redistribution or downstream use.
|
||||
|
||||
## Reproducibility
|
||||
|
||||
- SFT checkpoint: `sft-d24-v5-simplechat/iter_0001777`
|
||||
- HF export: `d24-sft-v5-simplechat-hf`
|
||||
- Chat template: `src/nemotron/data_prep/templates/simple_chatml.jinja`
|
||||
- Training repository: `yudasong/unified-training`
|
||||
1
chat_template.jinja
Normal file
1
chat_template.jinja
Normal file
@@ -0,0 +1 @@
|
||||
{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{'<|im_start|>assistant\n'}}{% endif %}
|
||||
29
config.json
Normal file
29
config.json
Normal file
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"architectures": [
|
||||
"LlamaForCausalLM"
|
||||
],
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"bos_token_id": 50256,
|
||||
"dtype": "float32",
|
||||
"eos_token_id": 50256,
|
||||
"head_dim": 128,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 1536,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 4096,
|
||||
"max_position_embeddings": 2048,
|
||||
"mlp_bias": false,
|
||||
"model_type": "llama",
|
||||
"num_attention_heads": 12,
|
||||
"num_hidden_layers": 24,
|
||||
"num_key_value_heads": 12,
|
||||
"pretraining_tp": 1,
|
||||
"rms_norm_eps": 1e-05,
|
||||
"rope_scaling": null,
|
||||
"rope_theta": 10000.0,
|
||||
"tie_word_embeddings": true,
|
||||
"transformers_version": "4.57.1",
|
||||
"use_cache": true,
|
||||
"vocab_size": 50304
|
||||
}
|
||||
9
generation_config.json
Normal file
9
generation_config.json
Normal file
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"_from_model_config": true,
|
||||
"bos_token_id": 50256,
|
||||
"eos_token_id": 50256,
|
||||
"output_attentions": false,
|
||||
"output_hidden_states": false,
|
||||
"transformers_version": "5.3.0",
|
||||
"use_cache": true
|
||||
}
|
||||
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:d847e1d6680e7a63510f70bcfcb6d14b6dd35f3bfdd4c8e524ac39af62d0d6fd
|
||||
size 1513663904
|
||||
250306
tokenizer.json
Normal file
250306
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
14
tokenizer_config.json
Normal file
14
tokenizer_config.json
Normal file
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"add_prefix_space": false,
|
||||
"backend": "tokenizers",
|
||||
"bos_token": "<|endoftext|>",
|
||||
"cls_token": "<|endoftext|>",
|
||||
"eos_token": "<|endoftext|>",
|
||||
"errors": "replace",
|
||||
"is_local": false,
|
||||
"model_max_length": 2048,
|
||||
"pad_token": null,
|
||||
"sep_token": "<|endoftext|>",
|
||||
"tokenizer_class": "GPT2Tokenizer",
|
||||
"unk_token": "<|endoftext|>"
|
||||
}
|
||||
Reference in New Issue
Block a user