zwhe99/DeepMath-1.5B
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: zwhe99/DeepMath-1.5B
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-08-20 06:43:16 +08:00
commit 9323b31389
14 changed files with 566 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

42
.gitattributes vendored Normal file
View File

@@ -0,0 +1,42 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
tokenizer.json filter=lfs diff=lfs merge=lfs -text
assets/github-contamination-case.png filter=lfs diff=lfs merge=lfs -text
assets/github-data-sample.png filter=lfs diff=lfs merge=lfs -text
assets/github-difficulty.png filter=lfs diff=lfs merge=lfs -text
assets/github-domain.png filter=lfs diff=lfs merge=lfs -text
assets/github-main.png filter=lfs diff=lfs merge=lfs -text
assets/github-tsne.png filter=lfs diff=lfs merge=lfs -text

243
README.md Normal file
View File

@@ -0,0 +1,243 @@
---
license: mit
datasets:
- zwhe99/DeepMath-103K
language:
- en
metrics:
- accuracy
base_model:
- deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
tags:
- math
- reasoning
- rl
- qwen
- qwen2
model-index:
- name: DeepMath-1.5B
results:
- task:
type: text-generation
dataset:
name: MATH500
type: MATH500
metrics:
- type: pass@1
value: 0.899
name: pass@1
verified: false
- task:
type: text-generation
dataset:
name: AMC23
type: AMC23
metrics:
- type: pass@1
value: 0.823
name: pass@1
verified: false
- task:
type: text-generation
dataset:
name: OlympiadBench
type: OlympiadBench
metrics:
- type: pass@1
value: 0.618
name: pass@1
verified: false
- task:
type: text-generation
dataset:
name: MinervaMath
type: MinervaMath
metrics:
- type: pass@1
value: 0.425
name: pass@1
verified: false
- task:
type: text-generation
dataset:
name: AIME24
type: AIME24
metrics:
- type: pass@1
value: 0.373
name: pass@1
verified: false
- task:
type: text-generation
dataset:
name: AIME25
type: AIME25
metrics:
- type: pass@1
value: 0.308
name: pass@1
verified: false
---
# DeepMath-1.5B
<table>
<tr>
<td style="padding: 0;">
<a href="https://huggingface.co/datasets/zwhe99/DeepMath-103K">
<img src="https://img.shields.io/badge/Data-4d5eff?style=for-the-badge&logo=huggingface&logoColor=ffffff&labelColor" alt="Data">
</a>
</td>
<td style="padding: 0;">
<a href="https://huggingface.co/collections/zwhe99/deepmath-6816e139b7f467f21a459a9a">
<img src="https://img.shields.io/badge/Model-4d5eff?style=for-the-badge&logo=huggingface&logoColor=ffffff&labelColor" alt="Data">
</a>
</td>
<td style="padding: 0;">
<a href="https://github.com/zwhe99/DeepMath">
<img src="https://img.shields.io/badge/Code-000000?style=for-the-badge&logo=github&logoColor=white" alt="Code">
</a>
</td>
<td style="padding: 0;">
<a href="https://arxiv.org/abs/2504.11456">
<img src="https://img.shields.io/badge/arXiv-2504.11456-b31b1b.svg?style=for-the-badge" alt="arXiv">
</a>
</td>
</tr>
</table>
<!-- Provide a quick summary of what the model is/does. -->
DeepMath-1.5B is created by finetuning deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B on DeepMath-103K dataset via RL.
## 📖 Overview
**`DeepMath-103K`** is meticulously curated to push the boundaries of mathematical reasoning in language models. Key features include:
**1. Challenging Problems**: DeepMath-103K has a strong focus on difficult mathematical problems (primarily Levels 5-9), significantly raising the complexity bar compared to many existing open datasets.
<div align="center"> <img src="./assets/github-difficulty.png" width="90%"/>
<sub>Difficulty distribution comparison.</sub> </div>
**2. Data Diversity and Novelty**: DeepMath-103K spans a wide spectrum of mathematical subjects, including Algebra, Calculus, Number Theory, Geometry, Probability, and Discrete Mathematics.
<div align="center"> <img src="./assets/github-domain.png" width="50%"/>
<sub>Hierarchical breakdown of mathematical topics covered in DeepMath-103K.</sub></div>
The problems in DeepMath-103K are novel and unique, whereas many existing datasets are similar and overlap.
<div align="center"> <img src="./assets/github-tsne.png" width="70%"/>
<sub>Embedding distributions of different datasets.</sub></div>
**3. Rigorous Decontamination**: Built from diverse sources, DeepMath-103K underwent meticulous decontamination against common benchmarks using semantic matching. This minimizes test set leakage and promotes fair model evaluation.
<div align="center"> <img src="./assets/github-contamination-case.png" width="80%"/>
<sub>Detected contamination examples. Subtle conceptual overlaps can also be identified.</sub> </div>
**4. Rich Data Format**: Each sample in DeepMath-103K is structured with rich information to support various research applications:
<div align="center"> <img src="./assets/github-data-sample.png" width="90%"/>
<sub>An example data sample from DeepMath-103K.</sub> </div>
- **Question**: The mathematical problem statement.
- **Final Answer**: A reliably verifiable final answer, enabling robust rule-based reward functions for RL.
- **Difficulty**: A numerical score for difficulty-aware training or analysis.
- **Topic**: Hierarchical classification for topic-specific applications.
- **R1 Solutions**: Three distinct reasoning paths from DeepSeek-R1, valuable for supervised fine-tuning (SFT) or knowledge distillation.
## 📊Main Results
DeepMath serise models achieve many **SOTA** results on challenging math benchmarks:
<div align="center"> <img src="./assets/github-main.png" width="90%"/>
<sub>Math reasoning performance.</sub> </div>
## 🎯Quick Start
#### Environment Preparation
```shell
git clone --recurse-submodules https://github.com/zwhe99/DeepMath.git && cd DeepMath
conda create -y -n deepmath python=3.12.2 && conda activate deepmath
pip3 install ray[default]
pip3 install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu124
pip3 install flash-attn==2.7.4.post1 --no-build-isolation
pip3 install omegaconf==2.4.0.dev3 hydra-core==1.4.0.dev1 antlr4-python3-runtime==4.11.0 vllm==0.7.3
pip3 install math-verify[antlr4_11_0]==0.7.0 fire deepspeed tensorboardX prettytable datasets transformers==4.49.0
pip3 install -e verl
```
#### Evaluation
```shell
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 VLLM_ATTENTION_BACKEND=XFORMERS VLLM_USE_V1=1 VLLM_WORKER_MULTIPROC_METHOD=spawn python3 uni_eval.py \
--base_model zwhe99/DeepMath-Zero-7B \
--chat_template_name orz \
--system_prompt_name simplerl \
--output_dir \
--bf16 True \
--tensor_parallel_size 8 \
--data_id zwhe99/MATH \
--split math500 \
--max_model_len 32768 \
--temperature 0.6 \
--top_p 0.95 \
--n 16
```
#### Training
* Data Preparation
```shell
DATA_DIR=/path/to/your/data
python3 verl/examples/data_preprocess/deepmath_103k.py --local_dir $DATA_DIR
```
* Start Ray
```shell
# Head node (×1)
ray start --head --port=6379 --node-ip-address=$HEAD_ADDR --num-gpus=8
# Worker nodes (×7 or ×11)
ray start --address=$HEAD_ADDR:6379 --node-ip-address=$WORKER_ADDR --num-gpus=8
```
* Launch training at head node. See `scripts/train` for training scripts.
## 🙏 Acknowledgements
This work can not be done without the help of the following works:
- **[verl](https://github.com/volcengine/verl)**: A very fast reinforcement learning framework.
- **[Vivacem/MMIQC](https://huggingface.co/datasets/Vivacem/MMIQC)**: A mixture of question-response pairs extracted from Mathematics Stack Exchange pages.
- **[TIGER-Lab/WebInstructSub](https://huggingface.co/datasets/TIGER-Lab/WebInstructSub)**: Instruction data from MathStackExchange and ScienceStackExchange.
- **[AI-MO/NuminaMath-CoT](https://huggingface.co/datasets/AI-MO/NuminaMath-CoT)**: Approximately 860k math problems.
## 📚 Citation
```bibtex
@article{deepmath,
title={DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning},
author={He, Zhiwei and Liang, Tian and Xu, Jiahao and Liu, Qiuzhi and Chen, Xingyu and Wang, Yue and Song, Linfeng and Yu, Dian and Liang, Zhenwen and Wang, Wenxuan and Zhang, Zhuosheng and Wang, Rui and Tu, Zhaopeng and Mi, Haitao and Yu, Dong},
year={2025},
eprint={2504.11456},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2504.11456},
}
```

3
assets/github-contamination-case.png Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ee68cf8039b340bf7f966b5524ddf6807ac70256d20e7ffc55bc7f0025725c0f
size 496279

3
assets/github-data-sample.png Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c73b736a266f868ae7b065b2732911576d609c0612c724e7ba8fe86fa2409ed0
size 2863342

3
assets/github-difficulty.png Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0460f9f283958ac9f3024208eb498d978cd0b1cb7cad5dc93e7207f09c1d2b8c
size 256873

3
assets/github-domain.png Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c0c3699f642bd1b5a59686d22bc71cc9fb0c4564154e142a8b26eb0abff6181e
size 2688729

3
assets/github-main.png Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a5e580dcd1607eec727bb971d20fddcc6fe55e53c252b2375a2904bf11b1a2b9
size 654091

3
assets/github-tsne.png Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4e2991c03a9481762a7b56369d3b479115644f98442eb09b3765d1491dad6013
size 3062689

32
config.json Normal file
View File

@@ -0,0 +1,32 @@
{
"architectures": [
"Qwen2ForCausalLM"
],
"attention_dropout": 0.0,
"bos_token_id": 151646,
"embd_pdrop": 0.0,
"eos_token_id": 151643,
"hidden_act": "silu",
"hidden_size": 1536,
"initializer_range": 0.02,
"intermediate_size": 8960,
"max_position_embeddings": 131072,
"max_window_layers": 21,
"model_type": "qwen2",
"num_attention_heads": 12,
"num_hidden_layers": 28,
"num_key_value_heads": 2,
"pad_token_id": 151643,
"resid_pdrop": 0.0,
"rms_norm_eps": 1e-06,
"rope_scaling": null,
"rope_theta": 10000,
"sliding_window": 4096,
"tie_word_embeddings": false,
"torch_dtype": "bfloat16",
"transformers_version": "4.49.0",
"use_cache": true,
"use_mrope": false,
"use_sliding_window": false,
"vocab_size": 151936
}

7
generation_config.json Normal file
View File

@@ -0,0 +1,7 @@
{
"_from_model_config": true,
"bos_token_id": 151646,
"eos_token_id": 151643,
"pad_token_id": 151643,
"transformers_version": "4.49.0"
}

3
model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f18856e41001b211c4435ac426c218f416a672cfc59155d35faf253632cb2906
size 3554214752

23
special_tokens_map.json Normal file
View File

@@ -0,0 +1,23 @@
{
"bos_token": {
"content": "<｜begin▁of▁sentence｜>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"eos_token": {
"content": "<｜end▁of▁sentence｜>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"pad_token": {
"content": "<｜end▁of▁sentence｜>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
}
}

3
tokenizer.json Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e20ddafc659ba90242154b55275402edeca0715e5dbb30f56815a4ce081f4893
size 11422778

195
tokenizer_config.json Normal file
View File

@@ -0,0 +1,195 @@
{
"add_bos_token": true,
"add_eos_token": false,
"add_prefix_space": null,
"added_tokens_decoder": {
"151643": {
"content": "<｜end▁of▁sentence｜>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151644": {
"content": "<｜User｜>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151645": {
"content": "<｜Assistant｜>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151646": {
"content": "<｜begin▁of▁sentence｜>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151647": {
"content": "<|EOT|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151648": {
"content": "<think>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151649": {
"content": "</think>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151650": {
"content": "<|quad_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151651": {
"content": "<|quad_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151652": {
"content": "<|vision_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151653": {
"content": "<|vision_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151654": {
"content": "<|vision_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151655": {
"content": "<|image_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151656": {
"content": "<|video_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151657": {
"content": "<tool_call>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151658": {
"content": "</tool_call>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151659": {
"content": "<|fim_prefix|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151660": {
"content": "<|fim_middle|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151661": {
"content": "<|fim_suffix|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151662": {
"content": "<|fim_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151663": {
"content": "<|repo_name|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151664": {
"content": "<|file_sep|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
}
},
"bos_token": "<｜begin▁of▁sentence｜>",
"chat_template": "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set ns = namespace(is_first=false, is_tool=false, is_output_first=true, system_prompt='') %}{%- for message in messages %}{%- if message['role'] == 'system' %}{% set ns.system_prompt = message['content'] %}{%- endif %}{%- endfor %}{{bos_token}}{{ns.system_prompt}}{%- for message in messages %}{%- if message['role'] == 'user' %}{%- set ns.is_tool = false -%}{{'<｜User｜>' + message['content']}}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is none %}{%- set ns.is_tool = false -%}{%- for tool in message['tool_calls']%}{%- if not ns.is_first %}{{'<｜Assistant｜><｜tool▁calls▁begin｜><｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{%- set ns.is_first = true -%}{%- else %}{{'\\n' + '<｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{{'<｜tool▁calls▁end｜><｜end▁of▁sentence｜>'}}{%- endif %}{%- endfor %}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is not none %}{%- if ns.is_tool %}{{'<｜tool▁outputs▁end｜>' + message['content'] + '<｜end▁of▁sentence｜>'}}{%- set ns.is_tool = false -%}{%- else %}{% set content = message['content'] %}{% if '</think>' in content %}{% set content = content.split('</think>')[-1] %}{% endif %}{{'<｜Assistant｜>' + content + '<｜end▁of▁sentence｜>'}}{%- endif %}{%- endif %}{%- if message['role'] == 'tool' %}{%- set ns.is_tool = true -%}{%- if ns.is_output_first %}{{'<｜tool▁outputs▁begin｜><｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- set ns.is_output_first = false %}{%- else %}{{'\\n<｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- endif %}{%- endif %}{%- endfor -%}{% if ns.is_tool %}{{'<｜tool▁outputs▁end｜>'}}{% endif %}{% if add_generation_prompt and not ns.is_tool %}{{'<｜Assistant｜><think>\\n'}}{% endif %}",
"clean_up_tokenization_spaces": false,
"eos_token": "<｜end▁of▁sentence｜>",
"extra_special_tokens": {},
"legacy": true,
"model_max_length": 16384,
"pad_token": "<｜end▁of▁sentence｜>",
"sp_model_kwargs": {},
"tokenizer_class": "LlamaTokenizerFast",
"unk_token": null,
"use_default_system_prompt": false
}