初始化项目，由ModelHub XC社区提供模型
Model: WeiboAI/VibeThinker-3B Source: Original Platform
This commit is contained in:
ModelHub XC
175
README.md
Normal file
175
README.md
Normal file
@@ -0,0 +1,175 @@
|
||||
---
|
||||
license: mit
|
||||
language:
|
||||
- en
|
||||
base_model:
|
||||
- Qwen/Qwen2.5-Coder-3B
|
||||
tags:
|
||||
- math
|
||||
- code
|
||||
- reasoning
|
||||
- gpqa
|
||||
- instruction-following
|
||||
pipeline_tag: text-generation
|
||||
library_name: transformers
|
||||
---
|
||||
|
||||
# VibeThinker-3B
|
||||
|
||||
<blockquote style="border-left: 4px solid #ff6b6b; background-color: #fff5f5; padding: 10px 15px; margin: 10px 0; color: #cc3333;">
|
||||
<span style="font-weight: bold;">🚨 </span> 1.This model was not trained on tool-calling or agent-based programming data. We therefore do not recommend using it for tasks that involve function calling, API orchestration, or autonomous coding agents.
|
||||
For programming tasks, we recommend using this model on competitive programming problems (e.g., <a href="https://leetcode.com/problemset/algorithms/">LeetCode-style</a>).
|
||||
|
||||
<br>
|
||||
<br>
|
||||
2.For harder math reasoning, try <a href="https://huggingface.co/datasets/meituan-longcat/AMO-Bench">AMOBench</a>, a problem set harder than the International Mathematical Olympiad (IMO), with included standard answers. Use it to evaluate VibeThinker against other SOTA models. Note: due to extreme difficulty, set max tokens to 60K–100K.
|
||||
|
||||
</blockquote>
|
||||
|
||||
<p align="center"><a href="https://github.com/WeiboAI/VibeThinker">GitHub</a> | <a href="https://modelscope.cn/models/WeiboAI/VibeThinker-3B">ModelScope</a> | <a href="https://huggingface.co/papers/2606.16140">Technical Report</a></p>
|
||||
|
||||
## Introduction
|
||||
|
||||
VibeThinker-3B is a further exploration of the VibeThinker series at the 3B-parameter scale, focusing on challenging reasoning tasks with clear verification signals, such as mathematics, coding, and STEM. By systematically optimizing the Spectrum-to-Signal Principle (SSP) post-training pipeline introduced in VibeThinker-1.5B, VibeThinker-3B achieves strong performance on AIME, HMMT, IMO-AnswerBench, LiveCodeBench, and recent LeetCode contests, reaching the performance range of top-tier frontier reasoning models, including Qwen3.6 Plus, Gemini 3 Pro, GLM-5, and Kimi K2.5, on verifiable reasoning benchmarks.
|
||||
|
||||
Motivated by these observations, we propose the Parametric Compression-Coverage Hypothesis: different capabilities depend on model parameters in fundamentally different ways. Verifiable reasoning is closer to a highly compressible, parameter-dense capability, centered on multi-step reasoning, constraint satisfaction, self-correction, and answer verification. When the task space is sufficiently structured and feedback signals are sufficiently reliable, compact models may also carry near-frontier reasoning capabilities. In contrast, open-domain knowledge, general-purpose dialogue, and long-tail scenario understanding rely more heavily on large-scale parameters to broadly cover facts, concepts, and world knowledge.
|
||||
|
||||
From VibeThinker-1.5B to VibeThinker-3B, our goal is not to build a small model that replaces large-scale models, but to examine the real boundaries of small models along specific capability dimensions. With VibeThinker-3B, we aim to show that small models should not be viewed merely as a compromise for reducing deployment costs. For capability domains with clear feedback and verification mechanisms, SLMs emerge as a promising research trajectory toward frontier-level performance that is fundamentally complementary to the traditional parameter scaling paradigm.
|
||||
|
||||
|
||||
|
||||
## Key Performance Data
|
||||
|
||||
📏 In terms of reasoning accuracy relative to model scale, VibeThinker-3B reaches 76.4 on IMO-AnswerBench, a highly challenging benchmark with 400 IMO-level problems, with only 3B parameters, and improves to 80.6 with Claim-Level Reliability Assessment (CLR), a test-time scaling strategy for answer-verifiable reasoning tasks. This demonstrates that a model within a strictly small-model regime can reach the performance range of substantially larger models, such as DeepSeek V3.2 (78.3, 671B), GLM-5 (82.5, 744B), and Kimi K2.5 (81.8, 1T).
|
||||
|
||||
|
||||
|
||||
💡 VibeThinker-3B achieves strong results across mathematics, coding, knowledge, and instruction-following benchmarks.
|
||||
|
||||
|
||||
|
||||
🔁 VibeThinker-3B achieves competitive results against first-tier reasoning models and reaches the performance range of top-tier systems on several verifiable reasoning benchmarks.
|
||||
|
||||
|
||||
|
||||
🏆 To further test the model's out-of-distribution performance, we evaluate VibeThinker-3B on recent unseen LeetCode weekly and biweekly contests (Python) from Apr. 25 to May 31, 2026. VibeThinker-3B passes **123/128** first-attempt submissions, corresponding to a **96.1%** acceptance rate.
|
||||
|
||||
|
||||
|
||||
|
||||
## Training Pipeline
|
||||
|
||||
VibeThinker-3B follows the **Spectrum-to-Signal Principle (SSP)** introduced in VibeThinker-1.5B. The SFT stage constructs a broad spectrum of valid reasoning trajectories, while the RL stage amplifies correct reasoning signals using verifiable rewards.
|
||||
|
||||
|
||||
|
||||
The training pipeline contains the following stages:
|
||||
|
||||
1. **Curriculum-based two-stage SFT**
|
||||
- Stage 1 focuses on broad capability coverage across math, code, STEM reasoning, general dialogue, and instruction following.
|
||||
- Stage 2 shifts toward harder and longer-horizon reasoning samples.
|
||||
- Diversity-Exploring Distillation is used to preserve multiple valid solution paths.
|
||||
|
||||
2. **Multi-domain Reasoning RL**
|
||||
- VibeThinker-3B reuses MaxEnt-Guided Policy Optimization (MGPO).
|
||||
- RL is applied sequentially to math, code, and STEM reasoning tasks.
|
||||
- Training uses a single 64K long-context window to preserve complete long-horizon reasoning trajectories.
|
||||
|
||||
3. **Offline Self-Distillation**
|
||||
- High-quality trajectories from Math, Code, and STEM RL checkpoints are filtered and distilled back into a unified student model.
|
||||
- A learning-potential score is used to prioritize traces that are correct but not yet well modeled by the student.
|
||||
|
||||
4. **Instruct RL**
|
||||
- The final stage improves controllability on user-facing prompts.
|
||||
- Rule-based validators and rubric-based reward models are used for format-sensitive and open-ended instruction data.
|
||||
|
||||
## Usage Guidelines
|
||||
|
||||
We recommend using VibeThinker-3B for competitive-style math, coding, STEM reasoning, and other tasks where the target answer can be verified. For broad open-domain knowledge tasks, larger general-purpose models may still be more suitable.
|
||||
|
||||
For benchmark-style evaluation, the technical report uses vLLM with:
|
||||
|
||||
- `temperature=1.0`
|
||||
- `top_p=0.95`
|
||||
- `top_k=-1`
|
||||
|
||||
## Quick Start
|
||||
|
||||
Required: **transformers>=4.54.0**
|
||||
|
||||
Recommended for better inference performance: **vLLM==0.10.1 or SGLang>=0.4.9.post6**
|
||||
|
||||
```python
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig
|
||||
|
||||
|
||||
class VibeThinker:
|
||||
def __init__(self, model_path):
|
||||
self.model_path = model_path
|
||||
self.model = AutoModelForCausalLM.from_pretrained(
|
||||
self.model_path,
|
||||
low_cpu_mem_usage=True,
|
||||
torch_dtype="bfloat16",
|
||||
device_map="auto",
|
||||
)
|
||||
self.tokenizer = AutoTokenizer.from_pretrained(
|
||||
self.model_path,
|
||||
trust_remote_code=True,
|
||||
)
|
||||
|
||||
def infer_text(self, prompt):
|
||||
messages = [{"role": "user", "content": prompt}]
|
||||
text = self.tokenizer.apply_chat_template(
|
||||
messages,
|
||||
tokenize=False,
|
||||
add_generation_prompt=True,
|
||||
)
|
||||
model_inputs = self.tokenizer([text], return_tensors="pt").to(self.model.device)
|
||||
|
||||
generation_config = dict(
|
||||
max_new_tokens=102400,
|
||||
do_sample=True,
|
||||
temperature=1.0,
|
||||
top_p=0.95,
|
||||
top_k=None,
|
||||
)
|
||||
generated_ids = self.model.generate(
|
||||
**model_inputs,
|
||||
generation_config=GenerationConfig(**generation_config),
|
||||
)
|
||||
generated_ids = [
|
||||
output_ids[len(input_ids):]
|
||||
for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
|
||||
]
|
||||
|
||||
return self.tokenizer.batch_decode(
|
||||
generated_ids,
|
||||
skip_special_tokens=True,
|
||||
)[0]
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
model = VibeThinker("WeiboAI/VibeThinker-3B")
|
||||
prompt = "Your Prompt"
|
||||
print(model.infer_text(prompt))
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
The model repository is licensed under the MIT License.
|
||||
|
||||
## Citations & References
|
||||
|
||||
If you use VibeThinker-3B in your research or product, please cite:
|
||||
|
||||
```bibtex
|
||||
@misc{xu2026vibethinker3bexploringfrontierverifiable,
|
||||
title={VibeThinker-3B: Exploring the Frontier of Verifiable Reasoning in Small Language Models},
|
||||
author={Sen Xu and Shixi Liu and Wei Wang and Jixin Min and Yingwei Dai and Zhibin Yin and Yirong Chen and Xin Zhou and Junlin Zhang},
|
||||
year={2026},
|
||||
eprint={2606.16140},
|
||||
archivePrefix={arXiv},
|
||||
primaryClass={cs.AI},
|
||||
url={https://arxiv.org/abs/2606.16140},
|
||||
}
|
||||
```
|
||||
Reference in New Issue
Block a user