Fig. 3: Analysis of representational formats using a feedforward deep neural network.

A Top: Representations in the feedforward network AlexNet. Representational Similarity Matrices (RSMs) reflecting pairwise correlations of unit activations in each layer of the network. Bottom: 2D Multidimensional Scaling (MDS) projections of RSMs at each layer, color-coded according to categories. B Representational consistency plot showing pairwise correlations (Spearman’s rho) of RSMs at each network layer. C Within-category, between-category and within-category vs. between-category correlations (i.e., Category Cluster Index, CCI) as a function of network layer. D Top: Correlations between RSMs from the DNN and neural data, for each AlexNet layer and each encoding time-frequency window in the VVS. Each time-frequency plot shows the correlation values of representations in one particular layer to neural representations. Clusters outlined in black indicate time-frequency periods where correlation values are significantly higher than zero at the group level (two-sided t-tests, Bonferroni corrected for 8 layers). Bottom: Same analysis for PFC data. Time zero in all panels indicates the onset of stimulus presentation E No matching of VVS RSMs with AlexNet RSMs during the maintenance period. Time zero indicates the onset of the cue. F Same analysis as in E for the PFC data. Color scale of all t-maps in F and G is indicated at the right of each panel. Source data are provided as a Source Data file. ***p < 0.001.