初始化项目,由ModelHub XC社区提供模型

Model: saidutta69/RaceBench-MiniCPM5
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-09-02 22:56:22 +08:00
commit 2f1e44a8b0
20 changed files with 654372 additions and 0 deletions

47
.gitattributes vendored Normal file
View File

@@ -0,0 +1,47 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
gguf/RaceBench-MiniCPM5-F16.gguf filter=lfs diff=lfs merge=lfs -text
gguf/RaceBench-MiniCPM5-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
gguf/RaceBench-MiniCPM5-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
gguf/RaceBench-MiniCPM5-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
gguf/RaceBench-MiniCPM5-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
gguf/RaceBench-MiniCPM5-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
gguf/RaceBench-MiniCPM5-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
gguf/RaceBench-MiniCPM5-Q5_0.gguf filter=lfs diff=lfs merge=lfs -text
gguf/RaceBench-MiniCPM5-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
gguf/RaceBench-MiniCPM5-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
gguf/RaceBench-MiniCPM5-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
gguf/RaceBench-MiniCPM5-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text

95
README.md Normal file
View File

@@ -0,0 +1,95 @@
---
license: apache-2.0
base_model: openbmb/MiniCPM5-1B
language:
- en
tags:
- fine-tuning
- racebench
- edge
- reasoning
- llm
pretty_name: RaceBench-MiniCPM5
pipeline_tag: text-generation
---
# RaceBench-MiniCPM5
<div align="center">
<img src="https://res.cloudinary.com/cmazqjs6/image/upload/racer_is_op_banner_branded_pu7zud.png" alt="RACER IS OP" width="100%">
</div>
Full-parameter fine-tune of [openbmb/MiniCPM5-1B](https://huggingface.co/openbmb/MiniCPM5-1B) on [**RaceBench**](https://huggingface.co/datasets/saidutta69/RaceBench), released as the reference checkpoint for the dataset - with fp16 weights and all GGUF quantizations (Q2_K through F16).
**The honest headline:** RaceBench transfers real multi-step reasoning gains to a 1B model (BBH +2.7, z=3.5), at the cost of a real, larger regression in math (GSM8K -9.9, z=-5.3). This is a tradeoff, not a win - and the tradeoff is the point. v2 of RaceBench adds formal-math data to close the math gap.
## Results: RaceBench-MiniCPM5 vs base
Evaluated with [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) v0.4.12, identical settings for both models (fp16, fixed seed, 2x Tesla T4). Standard errors reported per task; z = delta / pooled SE. |z| > 2 is statistically significant at ~95% confidence.
| Task | RaceBench-MiniCPM5 | MiniCPM5-1B (base) | Delta | z |
|---|---|---|---|---|
| **BBH (zero-shot, 27 subtasks, n=6511)** | **0.3397 ± 0.0056** | 0.3123 ± 0.0055 | **+2.74** | **+3.5** |
| GSM8K (n=1319) | 0.3108 ± 0.0127 | 0.4102 ± 0.0135 | **-9.93** | **-5.3** |
| MMLU (subset, n=5700) | 0.5310 ± 0.0060 | 0.5497 ± 0.0059 | -1.87 | -2.2 |
| Minerva MATH-500 (n=500) | 0.2140 ± 0.0184 | 0.2620 ± 0.0197 | -4.80 | -1.8 |
| Winogrande (n=1267) | 0.5391 ± 0.0140 | 0.5620 ± 0.0139 | -2.29 | -1.2 |
| HellaSwag (n=10042) | 0.4820 ± 0.0050 | 0.4881 ± 0.0050 | -0.61 | -0.9 |
| TruthfulQA MC2 (n=817) | 0.4710 ± 0.0149 | 0.4597 ± 0.0149 | +1.13 | +0.5 |
| GPQA Main n-shot (n=448) | 0.2589 ± 0.0207 | 0.2723 ± 0.0211 | -1.34 | -0.5 |
| ARC-Challenge (n=1176) | 0.3746 ± 0.0141 | 0.3831 ± 0.0142 | -0.85 | -0.4 |
### What is and isn't significant
- **Significant (|z| > 2):** BBH **+2.7** (reasoning gain), GSM8K **-9.9** (math loss), MMLU **-1.9** (knowledge loss)
- **Not significant (|z| < 2):** TruthfulQA, Winogrande, ARC, HellaSwag, MATH-500, GPQA - deltas within noise; no claim is made on these tasks
- **Bottom line:** RaceBench teaches multi-step reasoning (BBH) at the expense of math (GSM8K) and some knowledge recall (MMLU). The TruthfulQA and GPQA differences sometimes cited for this model are not statistically supported and should not be read as gains.
### Deployment implications (read before adopting)
- **Agents doing numeric reasoning (arithmetic, unit conversion, tool-call args): expect worse performance than the base model.** GSM8K is a direct proxy; the -9.9 is real and large.
- **Reasoning-heavy, math-light workloads (multi-hop analysis, planning, classification): expect measurable gains** - BBH +2.7 at z=3.5 is the most robust positive signal in this eval.
- **Not evaluated:** instruction-following (IFEval) and code generation (HumanEval, MBPP) were infeasible on T4 and are not reported. This checkpoint should not be marketed as an "edge agent" model until those are measured.
## Evaluation notes
- MMLU is a 5.7K-sample subset (100 per subtask) due to T4 memory limits; identical subset for both models. **It is not comparable to published full-MMLU scores** of other 1B models.
- Same seed, order, harness version for both models - deltas are head-to-head apples-to-apples, but all claims above require the significance test in the table.
- Single seed, single run per model. Significant effects (BBH, GSM8K, MMLU) survive the test; the rest should be treated as unknown, not as "retained competence."
## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"saidutta69/RaceBench-MiniCPM5",
trust_remote_code=True,
torch_dtype="float16",
)
tokenizer = AutoTokenizer.from_pretrained("saidutta69/RaceBench-MiniCPM5")
```
GGUF quants are in [`gguf/`](https://huggingface.co/saidutta69/RaceBench-MiniCPM5/tree/main/gguf) (F16, Q8_0, Q6_K, Q5_K_M, Q5_K_S, Q5_0, Q4_K_M, Q4_K_S, Q4_0, Q3_K_M, Q3_K_S, Q2_K).
## Training details
- **Base model:** openbmb/MiniCPM5-1B (Apache-2.0)
- **Dataset:** saidutta69/RaceBench (agent traces + quality-filtered coding/security/distilled)
- **Method:** full-parameter fine-tuning, fp16
## License
Apache-2.0 (base model weights); RaceBench data is MIT.
## Citation
```bibtex
@misc{racebench-minicpm5,
author = {Sai Dutta},
title = {RaceBench-MiniCPM5 -- First Public Fine-Tune on RaceBench},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/saidutta69/RaceBench-MiniCPM5}}
}
```

179
chat_template.jinja Normal file
View File

@@ -0,0 +1,179 @@
{{- bos_token }}{%- if tools %}
{%- set tool_definitions %}
{{- "# Tools\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
{%- for tool in tools %}
{{- "\n" }}
{{- tool | tojson(ensure_ascii=False) }}
{%- endfor %}
{{- '\n</tools>\n\nTool usage guidelines:\n- You may call zero or more functions. If no function calls are needed, just answer normally and do not include any <function ... </function>.\n- When calling a function, return an XML object within <function ... </function> using:\n<function name="function-name"><param name="param-name">param-value</param></function>\n- param-value may be multi-line. If it contains <, & or newline characters, wrap it in a CDATA block: <param name="param-name"><![CDATA[...multi-line value...]]></param>' }}
{%- endset %}
{{- '<|im_start|>system\n' }}
{%- if messages[0].role == 'system' %}
{%- if '<tool_def_sep>' in messages[0].content %}
{{- messages[0].content.replace('<tool_def_sep>', tool_definitions) }}
{%- else %}
{{- messages[0].content + '\n\n' + tool_definitions }}
{%- endif %}
{%- else %}
{{- tool_definitions.lstrip() }}
{%- endif %}
{{- '<|im_end|>\n' }}
{%- else %}
{%- if messages[0].role == 'system' %}
{{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
{%- for message in messages[::-1] %}
{%- set index = (messages|length - 1) - loop.index0 %}
{%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
{%- set ns.multi_step_tool = false %}
{%- set ns.last_query_index = index %}
{%- endif %}
{%- endfor %}
{%- for message in messages %}
{%- if message.content is string %}
{%- set content = message.content %}
{%- else %}
{%- set content = '' %}
{%- endif %}
{%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
{{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
{%- elif message.role == "assistant" %}
{%- set reasoning_content = '' %}
{%- if message.reasoning_content is string %}
{%- set reasoning_content = message.reasoning_content %}
{%- else %}
{%- if '</think>' in content %}
{%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
{%- set content = content.split('</think>')[-1].lstrip('\n') %}
{%- endif %}
{%- endif %}
{%- if message.tool_calls %}
{%- set content_parts = content.split('<tool_sep>') %}
{%- set processed_content = content_parts[0] %}
{%- set tool_calls_count = message.tool_calls|length %}
{%- set tool_sep_count = content_parts|length - 1 %}
{%- set min_count = [tool_calls_count, tool_sep_count]|min %}
{%- for i in range(1, content_parts|length) %}
{%- set tool_index = i - 1 %}
{%- if tool_index < tool_calls_count %}
{%- set tool_call = message.tool_calls[tool_index] %}
{%- if tool_call.function %}
{%- set tool_call = tool_call.function %}
{%- endif %}
{%- set single_tool_xml %}
{{- '<function name="' ~ tool_call.name ~ '">' }}
{%- if tool_call.arguments %}
{%- set args_dict = tool_call.arguments %}
{%- for param_name, param_value in args_dict.items() %}
{{- '<param name="' ~ param_name ~ '">' }}
{%- if param_value is string and ('<' in param_value or '&' in param_value or '\n' in param_value) %}
{{- '<![CDATA[' + param_value + ']]>' }}
{%- else %}
{{- param_value }}
{%- endif %}
{{- '</param>' }}
{%- endfor %}
{%- endif %}
{{- '</function>' }}
{%- endset %}
{%- set processed_content = processed_content + single_tool_xml + content_parts[i] %}
{%- else %}
{%- set processed_content = processed_content + content_parts[i] %}
{%- endif %}
{%- endfor %}
{%- if tool_calls_count > tool_sep_count %}
{%- for remaining_index in range(tool_sep_count, tool_calls_count) %}
{%- set tool_call = message.tool_calls[remaining_index] %}
{%- if tool_call.function %}
{%- set tool_call = tool_call.function %}
{%- endif %}
{%- set remaining_tool_xml %}
{{- '<function name="' ~ tool_call.name ~ '">' }}
{%- if tool_call.arguments %}
{%- set args_dict = tool_call.arguments %}
{%- for param_name, param_value in args_dict.items() %}
{{- '<param name="' ~ param_name ~ '">' }}
{%- if param_value is string and ('<' in param_value or '&' in param_value or '\n' in param_value) %}
{{- '<![CDATA[' + param_value + ']]>' }}
{%- else %}
{{- param_value }}
{%- endif %}
{{- '</param>' }}
{%- endfor %}
{%- endif %}
{{- '</function>' }}
{%- endset %}
{%- set processed_content = processed_content + remaining_tool_xml %}
{%- endfor %}
{%- endif %}
{%- set content = processed_content %}
{%- endif %}
{%- if loop.index0 > ns.last_query_index %}
{%- if reasoning_content %}
{{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
{%- else %}
{{- '<|im_start|>' + message.role + '\n' + content }}
{%- endif %}
{%- else %}
{{- '<|im_start|>' + message.role + '\n' + content }}
{%- endif %}
{%- if message.tool_calls and not has_tool_sep %}
{%- for tool_call in message.tool_calls %}
{%- if (loop.first and content) or (not loop.first) %}
{{- '\n' }}
{%- endif %}
{%- if tool_call.function %}
{%- set tool_call = tool_call.function %}
{%- endif %}
{{- '<function name="' ~ tool_call.name ~ '">' }}
{%- if tool_call.arguments %}
{%- set args_dict = tool_call.arguments %}
{%- for param_name, param_value in args_dict.items() %}
{{- '<param name="' ~ param_name ~ '">' }}
{%- if param_value is string and ('<' in param_value or '&' in param_value or '\n' in param_value) %}
{{- '<![CDATA[' + param_value + ']]>' }}
{%- else %}
{{- param_value }}
{%- endif %}
{{- '</param>' }}
{%- endfor %}
{%- endif %}
{{- '</function>' }}
{%- endfor %}
{%- endif %}
{{- '<|im_end|>\n' }}
{%- elif message.role == "tool" %}
{%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
{{- '<|im_start|>user' }}
{%- endif %}
{{- '\n<tool_response>\n' }}
{%- if message.content is string %}
{{- content }}
{%- else %}
{{- message.content | tojson(ensure_ascii=False) }}
{%- endif %}
{{- '\n</tool_response>' }}
{%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
{{- '<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
{{- '<|im_start|>assistant\n' }}
{%- if enable_thinking is defined %}
{%- if enable_thinking is false %}
{{- '<think>\n\n</think>\n\n' }}
{%- elif enable_thinking is true %}
{{- '<think>\n' }}
{%- endif %}
{%- endif %}
{%- endif %}

35
config.json Normal file
View File

@@ -0,0 +1,35 @@
{
"architectures": [
"LlamaForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 0,
"dtype": "float16",
"eos_token_id": [
1,
130073
],
"head_dim": 128,
"hidden_act": "silu",
"hidden_size": 1536,
"initializer_range": 0.02,
"intermediate_size": 4608,
"max_position_embeddings": 131072,
"mlp_bias": false,
"model_type": "llama",
"num_attention_heads": 16,
"num_hidden_layers": 24,
"num_key_value_heads": 2,
"pad_token_id": 1,
"pretraining_tp": 1,
"rms_norm_eps": 1e-06,
"rope_parameters": {
"rope_theta": 5000000,
"rope_type": "default"
},
"tie_word_embeddings": false,
"transformers_version": "5.14.1",
"use_cache": true,
"vocab_size": 130560
}

13
generation_config.json Normal file
View File

@@ -0,0 +1,13 @@
{
"_from_model_config": true,
"bos_token_id": 0,
"do_sample": true,
"eos_token_id": [
1,
130073
],
"pad_token_id": 1,
"temperature": 0.9,
"top_p": 0.95,
"transformers_version": "5.14.1"
}

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:73b15fdd6e61328c2c1cc37c4cc97de4bd905716e864d0dd07f945b035c9f7f6
size 2166551840

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:de92765ce06e0e87f1721263eedc132d6b5c1d13731fd61e4025d3472301b655
size 485829920

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:acbf7883c279eb2f40a5f718b4fde29e9a8ec2791bfadd3834a3461b4014c0ef
size 582898976

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:01bc2874f5a1f15351ec8718ae1ee104421bbac2909a9e076d2e4eeb1f3e3ff2
size 548074784

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:43c09151e6bcbedde4cf6055fba0f0e44de988aa45c2045f3bfa8074f7334029
size 664952096

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c8b8bd7860cc82651227d9d85d5152e2ba096f4a41f7df03965177b67137e795
size 688065824

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8a641b3329fc70238903417ab4ade499f2020557d68d4ed52e7844d85e9d5d15
size 667802912

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:046aa34bab3b2c77d73f258f34c4a3150533a1b6d1aa612a49c14f4657ea4c3b
size 774954272

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4f842d5ae87577c175d5fe8780546dc928b73646a8da6a20c704934ab200965f
size 786861344

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:12c7ecfa25c06183fe222e3d2c6e330f00f8e97f361855951b422b04233806df
size 774954272

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4a8b290441f28f9847b50aa73e663b3378a91565cc0da47a3da351a234bcba65
size 891831584

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:38d5e208f3c107b0f81ed1aa93d40f1f6e1436f26292b4476147ac7dcf974270
size 1153529120

3
model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0fcf983cb2350f4a500e9fee6eb96719f305dafe580c722e4aad983bc32e816a
size 2161290720

653947
tokenizer.json Normal file

File diff suppressed because it is too large Load Diff

17
tokenizer_config.json Normal file
View File

@@ -0,0 +1,17 @@
{
"add_prefix_space": null,
"backend": "tokenizers",
"bos_token": "<s>",
"clean_up_tokenization_spaces": false,
"eos_token": "</s>",
"is_local": false,
"legacy": true,
"local_files_only": false,
"model_max_length": 1000000000000000019884624838656,
"pad_token": "</s>",
"sp_model_kwargs": {},
"spaces_between_special_tokens": false,
"tokenizer_class": "TokenizersBackend",
"unk_token": "<unk>",
"use_default_system_prompt": false
}