Research
AutoRefine: Auto-Calibrating Code Judges from Workflow Outcomes
AutoRefine presents a loop that rewrites a deployable code-judge prompt to reproduce noisy outcome labels emitted by development workflows. An informed teacher sees the judge input plus the outcome, explains labels it retains, and abstains when the outcome cannot be explained from the judge's view. Calibration raised Scott's pi from 0.36 to 0.60 on a held-out dataset with an execution oracle, and recalibrating a deployed agent's judge raised agreement with merge/abandon outcomes from 0.16 to 0.51 on 17 independently held-out diffs. On older workflows, the previous judge remained ahead, illustrating the need for recalibration as the agent evolves.
AutoRefineLLM judgescode review