Back to AI News
arxiv

AURA Uses a Frozen General-Audio Codec for Spatial Audio-Language Tasks

AURA investigates whether a general-audio codec trained without spatial supervision can support both sound identification and localization in multi-source scenes. The authors find that spatial cues, including inter-channel differences, remain in the frozen codec but are not readily accessible to a language model due to audio-text information asymmetry. Their controlled experiments show that scene-description completion and inter-channel difference exposure each improve left/right accuracy, while binding directions to the queried source requires both. AURA keeps the codec and language model frozen and exceeds the strongest dedicated system on SpatialSoundQA by 10.31 points in dual-source direction accuracy using about 40% of its trainable parameters.

AI Brief

It presents a concrete spatial audio-language modeling approach with controlled analysis and a reported benchmark improvement over a dedicated system.

arXiv:2611.13687v1 Announce Type: cross Abstract: Answering spatial questions about multi-source scenes requires a model both to identify each sound and to locate it. Existing spatial audio-language models obtain spatial information from a dedicated encoder, which separates spatial and semantic processing into distinct feature spaces. We study whether a general-audio codec trained without spatial supervision can serve both purposes. A controlled layer-wise analysis shows that the frozen codec retains spatial information, with lateral cues carried by the difference between the two ear channels. This information, however, is not readily accessible: in dual-source scenes, a model trained on these features predicts left/right at chance and, on the remaining axes, assigns the detected direction to the queried and the non-queried source equally often. We attribute this failure to an audio--text information asymmetry, in which the audio contains both sources while the answer describes only the queried one. A controlled $2\times2$ experiment shows that either completing the scene description or exposing the inter-channel difference lifts left/right accuracy above chance, whereas binding directions to the queried source requires both. Building on these findings, AURA keeps the codec and the language model frozen and exceeds the strongest dedicated system on SpatialSoundQA by 10.31 points in dual-source direction accuracy, with about 40\% of its trainable parameters.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!