48 lines
2.2 KiB
Markdown
48 lines
2.2 KiB
Markdown
|
---
|
||
|
|
|
||
|
|
datasets: DeepMath-103k
|
||
|
|
library_name: transformers
|
||
|
|
model_name: TACReward7B
|
||
|
|
licence: license
|
||
|
|
---
|
||
|
|
|
||
|
|
<h1 align= "center"> Reasoning-Aware Proxy Reward Model using Process Mining </h1>
|
||
|
|
|
||
|
|
<p align="center">
|
||
|
|
<a href="https://pnubaelab.github.io/"><b>BAELAB</b></a>, Pusan National University, Busan, Korea
|
||
|
|
</p>
|
||
|
|
<p align="center">
|
||
|
|
<a href="https://yongzzai.com/">Yongjae Lee</a><sup>*</sup>, <a href="https://thrillcrazyer.github.io/">Taekyhun Park</a><sup>*</sup> , Hyerim Bae<sup>†</sup>
|
||
|
|
</p>
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
<p align="center">
|
||
|
|
<a href="https://github.com/Thrillcrazyer/TACReward"><b>🌟 Github</b></a> |
|
||
|
|
<a href="https://huggingface.co/Thrillcrazyer/Qwen-1.5B_THIP"><b>📥 1.5B Download</b></a> |
|
||
|
|
<a href="https://huggingface.co/Thrillcrazyer/TACReward7B"><b>📥 7B Download</b></a> |
|
||
|
|
<a href="https://arxiv.org/abs/2510.25065"><b>📄 Arxiv Paper Link</b></a> |
|
||
|
|
</p>
|
||
|
|
|
||
|
|
# Abstract
|
||
|
|
|
||
|
|
Recent advances in sparse reward policy gradient methods have enabled effective reinforcement learning (LR)
|
||
|
|
fine-tuning for post-training language models. However, for reasoning tasks such as mathematical problem solving,
|
||
|
|
binarized outcome rewards provide limited feedback on intermediate reasoning steps. While some studies have attempted
|
||
|
|
to address this issue by estimating **overall** reasoning quality, it remains unclear whether these rewards are
|
||
|
|
reliable proxies for the quality of stepwise reasoning. In this study, we consider reasoning as a structured process and
|
||
|
|
propose **TACReward** reward model. The model can be seamlessly integrated into sparse reward frameworks without
|
||
|
|
additional human annotation costs or architectural modifications. TACReward aggregates stepwise structural deviations
|
||
|
|
between teachers and policy reasoning using process mining techniques, producing a scalar output reward range of $[0, 1]$.
|
||
|
|
Experiments on multiple mathematical reasoning benchmarks demonstrate that integrating the TACReward into sparse reward
|
||
|
|
frameworks encourages the policy model to improve the structural quality of reasoning. Consequently, this leads to
|
||
|
|
consistent performance improvements over existing sparse reward frameworks.
|
||
|
|
|
||
|
|
# Illustration of TACReward
|
||
|
|
|
||
|
|
<div align="center">
|
||
|
|
<img src="https://arxiv.org/html/2510.25065v1/x1.png" width="600"/>
|
||
|
|
</div>
|