Live intelligence

Latest AI updates

1 updates
Research

DNN-Brain Alignment in Speech Processing Under Audiovisual Noise

This study examines how representations in a pretrained automatic speech recognition model align with human brain responses during acoustic-to-linguistic speech processing. Using neural encoding models and representational similarity analysis, the authors compare brain responses to acoustic and linguistic features with DNN embeddings before and after fine-tuning on audiovisual noisy data. Shallow layers show greater similarity to acoustic-related neural responses, while deeper layers align more closely with linguistic responses. Fine-tuning improves this alignment by enhancing noise processing in shallow layers and refining linguistic representations in deeper layers.

DNN-brain alignmentspeech processingaudiovisual noise