AIarXiv
Ranked by formulaVerdict: NotableCorrelation-Aware Exploration for Long-Trace Reasoning
Reasoning models trained with Group Relative Policy Optimization (GRPO) now produce chains of thought thousands of tokens long, and recent studies reveal that such training is prone to policy collapse.
Key numbers
- 8× fewer samples
- 4.17% on four backbones
VerdictWorth a reader's time today.
Abstract
Reasoning models trained with Group Relative Policy Optimization (GRPO) now produce chains of thought thousands of tokens long, and recent studies reveal that such training is prone to policy collapse. Entropy regularization is the standard remedy, but it works by pushing probability mass away from the few tokens that carry it and toward the vast low-probability tail of the vocabulary, and it applies this force once for every token of the reasoning trace. This puts the bonus in direct conflict with the entropy decay that convergence requires, and makes its coefficient difficult to set in the long-trace regime. To resolve this tension, we propose Info-GRPO, an information-theoretic framework that cultivates correlation between the policy and a latent prior. Rather than raising entropy everywhere, Info-GRPO builds on the mutual information between a latent variable and the trajectory it induces: augmenting prompts with latent seeds during training lets the model explore diverse policies correlated with the prior, while the seed-conditioned entropy is guided toward convergence. Diversity is therefore carried across whole reasoning traces rather than by per-token randomness, and exploration no longer fights entropy reduction. Experiments show that Info-GRPO outperforms vanilla GRPO and entropy-regularized GRPO across diverse reasoning benchmarks. On AIME24, it improves $Avg@8$ over GRPO by 3.75%, 1.66%, 4.16%, and 4.17% on four backbones, and it reaches the same majority-vote accuracy with 8× fewer samples. Code and models will be publicly released.
The editor's rubric
- Leverage4 / 5
- A general-purpose tool used across several fields (Adam, ResNet, LoRA, next-generation sequencing).
- Magnitude4 / 5
- A qualitative jump: a capability or regime that did not exist before.
- Evidence3 / 5
- Solid: multiple benchmarks or cohorts, ablations, fair baselines, released code or data.
- Novelty3 / 5
- A genuinely new approach to an open problem.
- Trajectory2 / 5
- Some room to improve with obvious engineering.
- Stakes2 / 5
- Benefits a professional community (practitioners, clinicians, engineers).
Editor’s rationale
Heuristic triage from title and abstract text only, not a reading of the paper. Cues found: method (we propose); breadth (many tasks); gains (x-fold, relative gain, outperforms); novelty (alternative to status quo); verification (code released); stakes (general AI).
How the score was computed
Desks place stronger verdicts first, then use this score to order work within the same verdict.
- Merit
- 6.3 / 10
- Adjusted merit
- 5.0 / 10
- Attention
- 0%
- Freshness
- 70%