122 lines
4.3 KiB
Markdown
122 lines
4.3 KiB
Markdown
---
|
||
license: apache-2.0
|
||
base_model:
|
||
- Qwen/Qwen3-8B
|
||
library_name: transformers
|
||
---
|
||
|
||
<div align="center">
|
||
|
||
# HIPO: Hybrid Policy Optimization for Dynamic Reasoning in LLMs
|
||
|
||
<img src="https://cdn-uploads.huggingface.co/production/uploads/61ee40a269351366e29972ad/KIYEa1c_WJEWPpeS0L_k1.png" width="60%" alt="Kwaipilot"/>
|
||
|
||
---
|
||
|
||
<a href="https://huggingface.co/Kwaipilot/HIPO-8B" target="_blank">
|
||
<img alt="Hugging Face" src="https://img.shields.io/badge/HuggingFace-fcd022?style=for-the-badge&logo=huggingface&logoColor=000&labelColor"/>
|
||
</a>
|
||
<a href="https://arxiv.org/abs/2509.23967" target="_blank">
|
||
<img alt="arXiv" src="https://img.shields.io/badge/arXiv-2509.23967-b31b1b.svg?style=for-the-badge"/>
|
||
</a>
|
||
|
||
<br>
|
||
|
||
<a href="https://arxiv.org/abs/2509.23967"></a>
|
||
|
||
</div>
|
||
|
||
This work is a companion to our earlier report [**HiPO: Hybrid Policy Optimization for Dynamic Reasoning in LLMs**](https://arxiv.org/abs/2509.23967), where we first introduced the **AutoThink paradigm** for controllable reasoning. While KAT-V1 outlined the overall framework of **SFT + RL** for adaptive reasoning, this paper provides the **detailed algorithmic design** of that training recipe.
|
||
|
||
***
|
||
|
||
# Overview
|
||
|
||
We introduce **HiPO (Hybrid Policy Optimization for Dynamic Reasoning in LLMs)**, a novel RL framework designed to enable models to decide when to “think” (i.e., Think-on)and when to skip reasoning (i.e., Think-off), thereby striking a balance between correctness and efficiency.
|
||
|
||
HIPO has two main components:
|
||
|
||
- **Hybrid Data Pipeline** – Collects both think-on and think-off responses, categorizes queries by difficulty, and uses a strong model (e.g., DeepSeek-V3) to generate explanations that justify mode choices.
|
||
- **Hybrid Reward System** – Combines rewards for both modes, with bias adjustment to prevent overuse of long reasoning and mode-aware advantage functions to align decisions with performance gains.
|
||
|
||
|
||
|
||
|
||
# Experimental Findings
|
||
|
||
**Think-on Only (Overthinking).**
|
||
Training only on Think-on data makes the model reason on all problems, causing inefficiency.
|
||
|
||
**GRPO.**
|
||
Improves accuracy by **+3.1%**, but increases token length on simple tasks.
|
||
|
||
**Think-on/Think-off Mix.**
|
||
Yields higher accuracy (**+4.0%**) while reducing token length (**–10.8%**) and thinking rate (**–22%**).
|
||
|
||
**HiPO Advantage.**
|
||
Achieves the best results: **+6.2% accuracy**, **–30% token length**, **–39% thinking rate**, outperforming existing methods in both **efficiency** and **accuracy**.
|
||
|
||
|
||
|
||
|
||
# Data Format
|
||
|
||
**HiPO** produces responses in a **structured template** that makes the reasoning path explicit and machine-parsable. Two modes are supported:
|
||
|
||
|
||
|
||
|
||
# Quick Start
|
||
|
||
```python
|
||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||
|
||
model_name = "Kwaipilot/HiPO-1.7B"
|
||
|
||
# load the tokenizer and the model
|
||
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
|
||
model = AutoModelForCausalLM.from_pretrained(
|
||
model_name,
|
||
torch_dtype="auto",
|
||
device_map="auto"
|
||
)
|
||
|
||
# prepare the model input
|
||
prompt = "Give me a short introduction to large language model."
|
||
messages = [
|
||
{"role": "user", "content": prompt}
|
||
]
|
||
text = tokenizer.apply_chat_template(
|
||
messages,
|
||
tokenize=False,
|
||
add_generation_prompt=True
|
||
)
|
||
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
|
||
|
||
# conduct text completion
|
||
generated_ids = model.generate(
|
||
**model_inputs,
|
||
max_new_tokens=32768,
|
||
temperature=0.6,
|
||
top_p=0.95,
|
||
)
|
||
output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist()
|
||
content = tokenizer.decode(output_ids, skip_special_tokens=True).strip("\n")
|
||
print("prompt:\n", prompt)
|
||
print("content:\n", content)
|
||
```
|
||
|
||
***
|
||
|
||
# Citation
|
||
|
||
```
|
||
@article{Zhan2025HiPO,
|
||
title={HiPO: Hybrid Policy Optimization for Dynamic Reasoning in LLMs},
|
||
author={Ken Deng, Zizheng Zhan, Wen Xiang, Wenqiang Zhu and others},
|
||
year={2025},
|
||
institution={arXiv preprint arXiv:2509.23967},
|
||
number={arXiv:2509.23967},
|
||
url={https://arxiv.org/abs/2509.23967}
|
||
}
|
||
``` |