AutoRefine: Auto-Calibrating Code Judges from Workflow Outcomes
AutoRefine presents a loop that rewrites a deployable code-judge prompt to reproduce noisy outcome labels emitted by development workflows. An informed teacher sees the judge input plus the outcome, explains labels it retains, and abstains when the outcome cannot be explained from the judge's view. Calibration raised Scott's pi from 0.36 to 0.60 on a held-out dataset with an execution oracle, and recalibrating a deployed agent's judge raised agreement with merge/abandon outcomes from 0.16 to 0.51 on 17 independently held-out diffs. On older workflows, the previous judge remained ahead, illustrating the need for recalibration as the agent evolves.
It offers a practical method for maintaining LLM code judges without fresh hand labels by using noisy workflow outcomes and abstention to flag when judge inputs are insufficient.
Comments & Questions (0)
No comments yet
Be the first to comment!