Research
Anytime-Valid Self-Consistency Certification Breaks Under Correlated LLM Reasoning
The paper shows that anytime-valid self-consistency certification for language models relies on an independence assumption that fails because reasoning rollouts are positively correlated through shared prefixes or prompt heterogeneity. Correlation splits the null hypothesis into a realized null about the sampled belief and a population null about average model behavior. The standard betting e-process remains valid only for the realized null; a prompt-level e-process restores population validity, with one rollout per prompt optimal under a pure rollout budget. Simulations and Claude Haiku 4.5 rollouts on GSM8K and AIME illustrate when the distinction matters.
LLM evaluationself-consistencystatistical reliability