The reasoning capabilities of advanced large language models (LLMs) like o1 have revolutionized artificial intelligence applications. Nevertheless, evaluating and optimizing complex reasoning processes remain significant challenges due to diverse policy distributions and the inherent limitations of human effort and accuracy. In this paper, we present AURORA, a novel automated framework for training universal process reward models (PRMs) using ensemble prompting and reverse verification. The framework employs a two-phase approach: First, it uses diverse prompting strategies and ensemble methods to perform automated annotation and evaluation of processes, ensuring robust assessments for reward learning. Second, it leverages practical reference answers for reverse verification, enhancing the model’s ability to validate outputs and improving training accuracy. To assess the framework’s performance, we extend beyond the existing ProcessBench benchmark by introducing UniversalBench, which evaluates reward predictions across full trajectories under diverse policy distribtion with long Chain-of-Thought (CoT) outputs. Experimental results demonstrate that AURORA enhances process evaluation accuracy, improves PRMs' accuracy for diverse policy distributions and long-CoT responses.

We evaluate the performance of the universal PRM trained using our proposed AURORA. First, we assess the effectiveness of our framework on ProcessBench, a benchmark specifically designed to evaluate generation processes using human-annotated labels. Next, we investigate the universal capabilities of the trained PRM across diverse policy distributions. To facilitate this evaluation, we construct a novel dataset called UniversalBench that spans a wide range of policy distributions, varying in both sequence length and step separation.

ProcessBench experimental results shown in Table 1 demonstrate that Universal-PRM-7B, trained using our proposed AURORA, achieves superior performance on the ProcessBench benchmark, excelling in both the overall average score and evaluations across four subsets. These results highlight the effectiveness of our approach in generalization under a universal policy training distribution and underscore the robustness of our proposed ensemble prompting techniques.

Experimental results, as shown in the Table 2, demonstrate that UniversalBench training under our proposed AURORA framework has achieved superior performance, highlighting its strong generalization capabilities across diverse policy distributions. By training under the AURORA framework, UniversalBench effectively addresses the challenges by constructing \(\mathcal{D}_{\text{gen}}\) using diverse policy and prompt distributions, particularly containing long CoT reasoning. This indicates the robustness of our approach in capturing and adapting to a wide range of policy behaviors, thereby outperforming existing methods in accuracy, making it applicable to real-world scenarios where the update policy distributions are dynamic.