Research
AuDiR: A Diagnostic Benchmark for Audio Reasoning with Controlled Interventions
AuDiR is a new benchmark that separates failures in acoustic perception, rule knowledge, and rule application for audio reasoning tasks. It uses controlled input interventions, progressively supplying rules and observations, and includes a no-audio condition. Preliminary results with Gemini-3.8-flash show accuracy jumps from 29.3% to 98.5% when observations are provided, highlighting the role of textual acoustic information. The benchmark includes 201 items across 18 families and 9 sources.
audio reasoningbenchmarkingmodel evaluation