Back to AI News
arxiv

AutoRefine: Auto-Calibrating Code Judges from Workflow Outcomes

AutoRefine presents a loop that rewrites a deployable code-judge prompt to reproduce noisy outcome labels emitted by development workflows. An informed teacher sees the judge input plus the outcome, explains labels it retains, and abstains when the outcome cannot be explained from the judge's view. Calibration raised Scott's pi from 0.36 to 0.60 on a held-out dataset with an execution oracle, and recalibrating a deployed agent's judge raised agreement with merge/abandon outcomes from 0.16 to 0.51 on 17 independently held-out diffs. On older workflows, the previous judge remained ahead, illustrating the need for recalibration as the agent evolves.

AI Brief

It offers a practical method for maintaining LLM code judges without fresh hand labels by using noisy workflow outcomes and abstention to flag when judge inputs are insufficient.

arXiv:2611.09291v1 Announce Type: new Abstract: LLM judges increasingly serve as quality gates for code changes that have no test oracle, but a judge prompt must be calibrated for its task and recalibrated as the system it judges evolves, while hand labels for the calibration are resource intensive and go stale. Development workflows emit outcome labels for free (tests pass or fail, changes land or are abandoned), but these labels are noisy. We present AutoRefine, a loop that rewrites a deployable judge's prompt to reproduce outcome labels, with an informed teacher that sees the judge's input plus the outcome, explains each label it keeps, and abstains when the outcome cannot be explained from what the judge sees. On a held-out dataset with an execution oracle, calibration raised chance-corrected agreement (Scott's $\pi$) from 0.36 to 0.60. On a deployed agent's outcome labels, with 25\% flipped at random to introduce noise, the teacher set aside flipped \textsc{fail} labels at more than twice the rate of genuine ones, cutting overall label noise from 25\% to 20\%. Recalibrating that agent's judge using AutoRefine raised its agreement with merge/abandon outcomes on 17 independently held-out diffs from the current workflow ($\pi$ 0.16 to 0.51); on 16 diffs from older workflows, the previous judge stayed ahead, illustrating the need for recalibration as the agent evolves. By abstaining, the teacher not only removes noisy labels but also flags changes whose outcome the judge's input cannot explain, telling a team when a judge needs better inputs rather than a better prompt.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!