Fig. 1: Overall framework for comparing representations in DNNs and the auditory pathway.
From: Dissecting neural computations in the human auditory pathway using deep neural networks for speech
The architecture of a family of DNN models, HuBERT/Wav2Vec 2, is illustrated on the left. The auditory pathway is illustrated on the right, with highlighted areas indicating the locations of the recorded/simulated electrophysiology signals. The same natural speech stimuli were presented to both the human participants and the DNN models, and the internal activations of each DNN layer were extracted and aligned with the corresponding neural activity from each recording site in the auditory pathway. A ridge regression model was fitted to predict neural activity from time-windowed DNN representations, and the regression coefficient of determination R2 between the predicted and actual neural activity was used as a metric of prediction accuracy.
