Research
DNN-Brain Alignment in Speech Processing Under Audiovisual Noise
This study examines how representations in a pretrained automatic speech recognition model align with human brain responses during acoustic-to-linguistic speech processing. Using neural encoding models and representational similarity analysis, the authors compare brain responses to acoustic and linguistic features with DNN embeddings before and after fine-tuning on audiovisual noisy data. Shallow layers show greater similarity to acoustic-related neural responses, while deeper layers align more closely with linguistic responses. Fine-tuning improves this alignment by enhancing noise processing in shallow layers and refining linguistic representations in deeper layers.
DNN-brain alignmentspeech processingaudiovisual noise