Research
When Expert Disagreement Hurts: Prestige and Selectivity in LLM Decision Revision
This paper introduces a controlled two-pass audit to measure whether LLMs improve or harm their decisions when revising after disagreement, separating reversal counts from accuracy effects. Across five models and 1,989 evidence-grounded decisions in medicine, science, and law, no model met the selectivity bar under expert disagreement, and accuracy fell by up to 16.1 points. The expert label increased reversals for four of five models, but its accuracy impact ranged from a 5.2-point loss to a 2.4-point gain, showing reversal counts are a poor proxy for harm. The authors argue revision pipelines should be evaluated with paired, conditional metrics per model and protocol.
language model behaviorAI evaluationhuman-AI interaction