初始化项目，由ModelHub XC社区提供模型
Model: nanbeige/Nanbeige4-3B-Base Source: Original Platform
This commit is contained in:
ModelHub XC
52
.gitattributes
vendored
Normal file
52
.gitattributes
vendored
Normal file
@@ -0,0 +1,52 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
|
||||
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zstandard filter=lfs diff=lfs merge=lfs -text
|
||||
*.tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
*.db* filter=lfs diff=lfs merge=lfs -text
|
||||
*.ark* filter=lfs diff=lfs merge=lfs -text
|
||||
**/*ckpt*data* filter=lfs diff=lfs merge=lfs -text
|
||||
**/*ckpt*.meta filter=lfs diff=lfs merge=lfs -text
|
||||
**/*ckpt*.index filter=lfs diff=lfs merge=lfs -text
|
||||
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.gguf* filter=lfs diff=lfs merge=lfs -text
|
||||
*.ggml filter=lfs diff=lfs merge=lfs -text
|
||||
*.llamafile* filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
|
||||
tokenizer.model filter=lfs diff=lfs merge=lfs -text
|
||||
model.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
||||
pytorch_model.bin filter=lfs diff=lfs merge=lfs -text
|
||||
127
README.md
Normal file
127
README.md
Normal file
@@ -0,0 +1,127 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
language:
|
||||
- en
|
||||
- zh
|
||||
library_name: transformers
|
||||
pipeline_tag: text-generation
|
||||
tags:
|
||||
- llm
|
||||
- nanbeige
|
||||
---
|
||||
<div align="center">
|
||||
|
||||
<img src="figures/nbg.png" width="220" alt="Nanbeige Logo">
|
||||
</div>
|
||||
|
||||
|
||||
# <span id="Introduction">1. Introduction</span>
|
||||
|
||||
Nanbeige4-3B-Base is a **3B-parameter base model** within the fourth-generation Nanbeige LLM family.
|
||||
It showcases that even a compact model can achieve advanced performances through continuous enhancements in data quality and training methodologies.
|
||||
When performing supervised fine-tuning (SFT) on the same training data, our model significantly outperforms open-source models of the same size, and even surpasses larger models such as Qwen3-8B.
|
||||
|
||||
* Technical Report: https://arxiv.org/pdf/2512.06266
|
||||
|
||||
<div align="center">
|
||||
|
||||
<img src="figures/performance_base.png">
|
||||
</div>
|
||||
|
||||
# <span id="Model Summary">2. Model Summary</span>
|
||||
|
||||
Training Data<br>
|
||||
|
||||
* We constructed a comprehensive **23T-tokens** training corpus from web texts, books, code, and papers, meticulously filtered through a hybrid strategy of tagging-based scoring and retrieval-based recalling.
|
||||
This foundation was then augmented with **knowledge-dense and reasoning-intensive synthetic data**, including Q&A pairs, textbooks, and Long-COTs, which significantly benefited the downstream task performance.
|
||||
|
||||
|
||||
Training Recipe<br>
|
||||
|
||||
* We designed an innovative **FG-WSD (Fine-Grained Warmup-Stable-Decay)** training scheduler, meticulously refining the conventional WSD approach.
|
||||
This scheduler was implemented with a **fine-grained, quality-progressive data curriculum**, dividing the Stable stage into multiple phases with progressively improved data mixtures. Compared to the vanilla WSD, our method achieved notable performance gains. During the Decay stage, we increased the proportion of math, code, synthetic QA, and synthetic Long-COT data to further enhance reasoning capabilities.
|
||||
| Stage | Training Tokens | Learning Rate |
|
||||
|-------------------------------|-----------------|-----------------------|
|
||||
| Warmup Stage | 0.1T | 0 ——> 4.5e-4 |
|
||||
| Diversity-Enriched Stable Stage| 12.4T | Constant 4.5e-4 |
|
||||
| High-Quality Stable Stage | 6.5T | Constant 4.5e-4 |
|
||||
| Decay and Long-Context Stage | 4T | 4.5e-4 ——> 1.5e-6 |<br><br>
|
||||
|
||||
|
||||
|
||||
# <span id="Model Performance">3. Model Performance</span>
|
||||
|
||||
For model performance comparison, we fine-tuned both our base model and the Qwen series base models using the same fine-tuning data and evaluated their downstream task metrics. We believe that when evaluating base models, this end-to-end validation approach better reflects the model's ultimate performance in downstream tasks compared to the few-shot testing approach.
|
||||
|
||||
To ensure a fair comparison, we conducted experiments with three distinct datasets, including [Nemotron-Dataset-v1](https://huggingface.co/datasets/nvidia/Nemotron-Post-Training-Dataset-v1), [Ring-lite-sft-data](https://huggingface.co/datasets/inclusionAI/Ring-lite-sft-data), and [OpenThoughts3](https://huggingface.co/datasets/open-thoughts/OpenThoughts3-1.2M). For each dataset, we randomly select 500k training samples for SFT experiments.
|
||||
|
||||
* Finetuned with Nemotron-Dataset-v1
|
||||
| Model | AIME2024 | AIME2025 | Math-500 | GPQA |
|
||||
|--------------------|----------|----------|----------|------|
|
||||
| Qwen3-4B-Base | 24.6 | 25.0 | 90.4 | 44.6 |
|
||||
| Qwen3-8B-Base | <u>37.9<u> | <u>29.6<u> | <u>91.1<u> | <u>48.9<u> |
|
||||
| **Nanbeige4-3B-Base** | **52.9** | **40.8** | **93.4** | **53.4** |
|
||||
|
||||
* Finetuned with Ring-lite-sft-data
|
||||
| Model | AIME2024 | AIME2025 | Math-500 | GPQA |
|
||||
|--------------------|----------|----------|----------|------|
|
||||
| Qwen3-4B-Base | 40.4 | 31.3 | 93.6 | 51.4 |
|
||||
| Qwen3-8B-Base | <u>50.0<u> | <u>35.8<u> | <u>94.4<u> | <u>55.1<u> |
|
||||
| **Nanbeige4-3B-Base** | **56.8** | **45.3** | **95.5** | **57.7** |
|
||||
|
||||
* Finetuned with OpenThoughts3
|
||||
| Model | AIME2024 | AIME2025 | Math-500 | GPQA |
|
||||
|--------------------|----------|----------|----------|------|
|
||||
| Qwen3-4B-Base | 52.9 | 42.1 | 93.2 | 49.6 |
|
||||
| Qwen3-8B-Base | <u>60.4<u> | <u>47.1<u> | **95.0** | <u>55.3<u> |
|
||||
| **Nanbeige4-3B-Base** | **62.4** | **49.2** | <u>94.6<u> | **56.9** |
|
||||
|
||||
The results demonstrate that **Nanbeige4-3B-Base** significantly outperforms Qwen3-4B-Base, and even surpasses the larger Qwen3-8B-Base, highlighting the greater potential of our base model after fine-tuning. This advantage stems from the optimized training recipe during our Stable stage and the extensive high-quality synthetic data incorporated during the Decay stage.
|
||||
|
||||
|
||||
## <span id="Inference">4. Quickstart</span>
|
||||
|
||||
```
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
tokenizer = AutoTokenizer.from_pretrained(
|
||||
'Nanbeige/Nanbeige4-3B-Base',
|
||||
use_fast=False,
|
||||
trust_remote_code=True
|
||||
)
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
'Nanbeige/Nanbeige4-3B-Base',
|
||||
torch_dtype='auto',
|
||||
device_map='auto',
|
||||
trust_remote_code=True
|
||||
)
|
||||
prompt = "中国的首都是"
|
||||
input_ids = tokenizer(prompt, return_tensors='pt').input_ids
|
||||
output_ids = model.generate(input_ids.to('cuda'))
|
||||
resp = tokenizer.decode(output_ids[0][len(input_ids[0]):], skip_special_tokens=True)
|
||||
print(resp)
|
||||
```
|
||||
|
||||
|
||||
# <span id="Limitations">5. Limitations</span>
|
||||
|
||||
While we place great emphasis on the safety of the model during the training process, striving to ensure that its outputs align with ethical and legal requirements, it may not completely avoid generating unexpected outputs due to the model's size and probabilistic nature. These outputs may include harmful content such as bias or discrimination. Please don't propagate such content. We do not assume any responsibility for the consequences resulting from the dissemination of inappropriate information.
|
||||
<br>
|
||||
|
||||
# <span id="Limitations">6. Citation</span>
|
||||
If you find our model useful or want to use it in your projects, please cite as follows:
|
||||
```
|
||||
@misc{yang2025nanbeige43btechnicalreportexploring,
|
||||
title={Nanbeige4-3B Technical Report: Exploring the Frontier of Small Language Models},
|
||||
author={Chen Yang and Guangyue Peng and Jiaying Zhu and Ran Le and Ruixiang Feng and Tao Zhang and Wei Ruan and Xiaoqi Liu and Xiaoxue Cheng and Xiyun Xu and Yang Song and Yanzipeng Gao and Yiming Jia and Yun Xing and Yuntao Wen and Zekai Wang and Zhenwei An and Zhicong Sun and Zongchao Chen},
|
||||
year={2025},
|
||||
eprint={2512.06266},
|
||||
archivePrefix={arXiv},
|
||||
primaryClass={cs.CL},
|
||||
url={https://arxiv.org/abs/2512.06266},
|
||||
}
|
||||
```
|
||||
<br>
|
||||
|
||||
# <span id="Limitations">7. Contact</span>
|
||||
If you have any questions, please raise an issue or contact us at nanbeige@126.com.
|
||||
<br>
|
||||
7
added_tokens.json
Normal file
7
added_tokens.json
Normal file
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"</think>": 166104,
|
||||
"<think>": 166103,
|
||||
"<|endoftext|>": 166102,
|
||||
"<|im_end|>": 166101,
|
||||
"<|im_start|>": 166100
|
||||
}
|
||||
27
config.json
Normal file
27
config.json
Normal file
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"architectures": [
|
||||
"LlamaForCausalLM"
|
||||
],
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"bos_token_id": 1,
|
||||
"eos_token_id": 2,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 2560,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 10496,
|
||||
"max_position_embeddings": 32768,
|
||||
"model_type": "llama",
|
||||
"num_attention_heads": 20,
|
||||
"num_hidden_layers": 32,
|
||||
"num_key_value_heads": 4,
|
||||
"pretraining_tp": 1,
|
||||
"rms_norm_eps": 1e-05,
|
||||
"rope_scaling": null,
|
||||
"rope_theta": 5000000,
|
||||
"tie_word_embeddings": false,
|
||||
"torch_dtype": "bfloat16",
|
||||
"transformers_version": "4.38.2",
|
||||
"use_cache": true,
|
||||
"vocab_size": 166144
|
||||
}
|
||||
1
configuration.json
Normal file
1
configuration.json
Normal file
@@ -0,0 +1 @@
|
||||
{"framework": "pytorch", "task": "text-generation", "allow_remote": true}
|
||||
BIN
figures/nbg.png
Normal file
BIN
figures/nbg.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 15 KiB |
BIN
figures/performance_base.png
Normal file
BIN
figures/performance_base.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 129 KiB |
6
generation_config.json
Normal file
6
generation_config.json
Normal file
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"_from_model_config": true,
|
||||
"bos_token_id": 1,
|
||||
"eos_token_id": 2,
|
||||
"transformers_version": "4.38.2"
|
||||
}
|
||||
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:d7a6ede44a91b758b5da0ece5272c740587f0e483ec5d250c451cc6c33d84ad1
|
||||
size 7867308040
|
||||
3
pytorch_model.bin
Normal file
3
pytorch_model.bin
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:f0df581bc7823d94e4b8cbf7ccc53415bf6aef20e48e0374bfc49d2fd65cc549
|
||||
size 7867370154
|
||||
3
tokenizer.json
Normal file
3
tokenizer.json
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:34bdaf57c31aef664b6e62c3b4fbdcb49171508c8d38942413a92d4a28707866
|
||||
size 9840344
|
||||
3
tokenizer.model
Normal file
3
tokenizer.model
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:fb41d04798b714520a9b075727b0226538b7330254299062742c50ec8374bc36
|
||||
size 2782298
|
||||
61
tokenizer_config.json
Normal file
61
tokenizer_config.json
Normal file
@@ -0,0 +1,61 @@
|
||||
{
|
||||
"add_bos_token": true,
|
||||
"add_eos_token": false,
|
||||
"add_prefix_space": true,
|
||||
"added_tokens_decoder": {
|
||||
"0": {
|
||||
"content": "<unk>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"1": {
|
||||
"content": "<s>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"2": {
|
||||
"content": "</s>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"166103": {
|
||||
"content": "<think>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"166104": {
|
||||
"content": "</think>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
}
|
||||
},
|
||||
"additional_special_tokens": [
|
||||
"</think>"
|
||||
],
|
||||
"bos_token": "<s>",
|
||||
"clean_up_tokenization_spaces": false,
|
||||
"eos_token": "</s>",
|
||||
"legacy": true,
|
||||
"model_max_length": 1000000000000000019884624838656,
|
||||
"pad_token": null,
|
||||
"sp_model_kwargs": {},
|
||||
"spaces_between_special_tokens": false,
|
||||
"tokenizer_class": "LlamaTokenizer",
|
||||
"unk_token": "<unk>",
|
||||
"use_default_system_prompt": false
|
||||
}
|
||||
Reference in New Issue
Block a user