This April 2025 paper introduces Self-Principled Critique Tuning (SPCT), a novel method designed to enhance the inference-time scalability of Generative Reward Models (GRMs) for various domains. It details how SPCT, through a combination of rejective fine-tuning and rule-based online reinforcement learning, facilitates the adaptive generation of principles and critiques, thereby improving the quality and inference-time scalability of GRMs. The paper compares different reward generation paradigms (scalar, semi-scalar, generative) and scoring patterns (pointwise, pairwise), demonstrating that the proposed DeepSeek-GRM models, particularly when guided by a meta Reward Model, consistently outperform existing methods across multiple benchmarks without significant domain biases. The research highlights the potential for GRMs to serve as a versatile interface for generalist reward systems, advancing Large Language Model (LLM) post-training and inference, while also acknowledging ethical considerations regarding bias and the importance of human-in-the-loop frameworks.
Source:
https://arxiv.org/pdf/2504.02495
