Project Website | 📄 Paper | 🛠️ GitHub | 🤗 Dataset | 🤗 Models
This repository includes code and materials for the paper "Efficient PRM Training Data Synthesis via Formal Verification" (ACL 2026 Findings). * GitHub: [https://github.com/psunlpgroup/FoVer](https://github.com/psunlpgroup/FoVer) * FoVer PRMs * [ryokamoi/Llama-3.1-8B-FoVer-PRM-2026](https://huggingface.co/ryokamoi/Llama-3.1-8B-FoVer-PRM-2026) * [ryokamoi/Qwen-2.5-7B-FoVer-PRM-2026](https://huggingface.co/ryokamoi/Qwen-2.5-7B-FoVer-PRM-2026) ```bibtex @inproceedings{kamoi2026fover, title = {Efficient PRM Training Data Synthesis via Formal Verification}, author = {Ryo Kamoi and Yusen Zhang and Nan Zhang and Sarkar Snigdha Sarathi Das and Ranran Haoran Zhang and Wenpeng Yin and Rui Zhang}, year = {2026}, booktitle = {Findings of the Association for Computational Linguistics: ACL 2026}, } ``` ## Introduction Process Reward Models (PRMs) have emerged as a promising approach for improving LLM reasoning capabilities by providing process supervision over reasoning traces. However, existing approaches for constructing PRM training data remain costly and noisy, as they typically rely on human annotation or sampling-based labeling methods that require repeated LLM calls. In this work, we propose FoVer, a framework that synthesizes PRM training data from formal reasoning tasks by annotating step-level error labels using formal verification tools such as Z3 and Isabelle. By leveraging formal verification, FoVer enables efficient and accurate PRM data construction without requiring human annotation or additional LLM calls. Using FoVer, we create PRM training data from formal logic and theorem proving tasks. Experiments on 12 reasoning benchmarks show that fine-tuning on our training data improves PRMs not only on math and logic reasoning tasks, which are informal variants of the training tasks, but also on NLI and BBH benchmarks, which differ substantially from the tasks used to construct the training data. These results demonstrate the practical effectiveness of FoVer, showing that PRM training data created using formal verification improves PRMs on informal reasoning tasks written in natural language.