Back to AI News
arxiv

AuDiR: A Diagnostic Benchmark for Audio Reasoning with Controlled Interventions

AuDiR is a new benchmark that separates failures in acoustic perception, rule knowledge, and rule application for audio reasoning tasks. It uses controlled input interventions, progressively supplying rules and observations, and includes a no-audio condition. Preliminary results with Gemini-3.8-flash show accuracy jumps from 29.3% to 98.5% when observations are provided, highlighting the role of textual acoustic information. The benchmark includes 201 items across 18 families and 9 sources.

AI Brief

This introduces a novel diagnostic methodology for audio AI, offering actionable insights into model reasoning failures beyond simple accuracy.

arXiv:2611.12478v1 Announce Type: new Abstract: Answer accuracy alone does not distinguish failures in acoustic perception, rule knowledge, and rule application. AuDiR (Audio Diagnostic Reasoning) examines these distinctions through controlled input interventions. Each item contains an audio clip, a multiple-choice question, a task-general rule, and a textual description of the acoustic observations needed to answer. Three conditions progressively supply the rule and observations; a fourth removes the audio while retaining the question and rule. The first correct diagnostic condition assigns an operational response label, without identifying a unique internal cause. The current benchmark contains 201 items in 18 families drawn from 9 sources, with answer references documented separately from percept extraction. In a preliminary evaluation of Gemini-3.8-flash, accuracy is 29.3% without supplied information, 31.3% with the rule, and 98.5% with the observations also supplied. No-audio accuracy is 23.9%, with a confidence interval containing the 25.0% chance rate. A family-level knowledge probe associates incomplete rule recall with larger gains from rule provision. These results show the benefit of supplying acoustic observations as text, while incomplete validation, possible recognition shortcuts, and non-monotonic responses limit stronger failure-attribution claims.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!