DNN-Brain Alignment in Speech Processing Under Audiovisual Noise
This study examines how representations in a pretrained automatic speech recognition model align with human brain responses during acoustic-to-linguistic speech processing. Using neural encoding models and representational similarity analysis, the authors compare brain responses to acoustic and linguistic features with DNN embeddings before and after fine-tuning on audiovisual noisy data. Shallow layers show greater similarity to acoustic-related neural responses, while deeper layers align more closely with linguistic responses. Fine-tuning improves this alignment by enhancing noise processing in shallow layers and refining linguistic representations in deeper layers.
It provides evidence that fine-tuned speech models can develop brain-like hierarchical processing under noisy audiovisual conditions, which is useful for researchers studying model interpretability and human-aligned speech AI.
Comments & Questions (0)
No comments yet
Be the first to comment!