Fig. 6: Overt behavioural choices in humans and the DNN.
From: Dimensions underlying the representational alignment of deep neural networks with humans
a, Overview of the approach. For one triplet, we computed the original predicted softmax probability based on the entire representational embedding for each object image in the triplet. We then iteratively pruned individual dimensions from the representational embedding and stored the resulting change in the predicted softmax probability—relative to that of the full embedding—as a relevance score for that dimension. b, We calculated the relevance scores for a random sample of 10 million triplets and identified the most relevant dimension for each triplet. We then labelled the 10 million most relevant dimensions according to human-labelled visual properties as semantic, mixed visual–semantic, visual or unclear. Semantic dimensions are the most relevant for human behavioural choices, whereas for VGG-16, visual and mixed visual–semantic properties are more relevant. c–f, We rank the sorted changes in softmax probability to find triplets in which human and the DNN maximally diverge. Each panel shows a triplet with the behavioural choice made by humans and the DNN. We visualized the most relevant dimension for that triplet alongside the distribution of relevance scores. Each dimension is assigned its human-annotated label. For this figure, we filtered the embedding by images from the public domain76. Images in a and c–f reproduced with permission from ref. 76, Springer Nature Limited.
