Back to AI News
arxiv

AudioSphere: Self-Supervised Spatial Audio Representations for Frame-Level SELD

AudioSphere is a multi-channel masked autoencoder trained on Ambisonics recordings that reconstructs spectral content and directional cues using a joint spatiotemporal masking strategy over time-frequency patches and intensity vectors. The model supports frame-level predictions and produces clip-level representations through aggregation. The authors also release RealSELD, a standardized benchmark for evaluating frozen backbones on real-world multi-channel sound event detection and localization. They report that AudioSphere enables SELD with a frozen self-supervised backbone and matches or exceeds prior self-supervised audio embedding models on clip-level classification and frame-level sound event detection.

AI Brief

It introduces a spatial self-supervised audio backbone plus a real-world SELD benchmark, offering practical value for researchers working on multi-channel audio understanding.

arXiv:2611.09275v1 Announce Type: cross Abstract: Recent audio embedding models learn representations that transfer across clip- and frame-level tasks, yet they operate almost exclusively on single-channel signals and are blind to spatial structure. Spatial audio models capture that structure but rely on clip-level supervision or contrastive objectives, yielding embeddings unsuitable for frame-level tasks like sound event detection and localization (SELD). We argue this dichotomy is not fundamental: patch-level spatial self-supervision can jointly learn acoustic and spatial features at both temporal granularities. We introduce AudioSphere, a multi-channel masked autoencoder trained on Ambisonics recordings that reconstructs both spectral content and directional cues via a joint spatiotemporal masking strategy that masks both time-frequency patches and intensity vectors. AudioSphere supports frame-level predictions while producing strong clip-level representations via aggregation. Alongside the model, we release RealSELD, a standardized benchmark for evaluating frozen backbones on real-world multi-channel SELD. AudioSphere enables, for the first time, SELD with a frozen self-supervised backbone, while matching or exceeding prior self-supervised audio embedding models on clip-level classification tasks spanning environmental sound, speech, and music, and on frame-level sound event detection. All code and data are available at https://github.com/labhamlet/AudioSphere and https://huggingface.co/labhamlet.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!