Fig. 3: Understanding concepts and concept composition with CRP.
From: From attribution maps to human-understandable explanations through Concept Relevance Propagation
a, Given an input image for inference, a constitutes a traditional attribution map indicating that various body parts of the bird are relevant for the prediction. b, Channel-conditional explanations computed with CRP help to localize and understand channel concepts by providing masked reference samples (explaining by example with RelMax). c, CRP relevances can further be used to construct a concept atlas, visualizing which concepts dominate in specific regions in the input image defined by super-pixels. Here, the most relevant channels in layer layer3.0.conv2 can be identified with concepts ‘dots’ (channels 210 and 130), ‘red spot’ (10), ‘black eyes’ (187) and ‘stripes-like’ (19). d, Concept-composition graphs decompose a concept of interest given a particular prediction into lower-layer concepts, thus improving concept understanding. Shown are relevant (sub)-concepts in features.24 and features.26 for concept ‘animal on branch’ in features.28 for the prediction of class ‘Bee Eater’. The relevance flow is highlighted in red, with the relative percentage of relevance flow to the lower-level concepts. For each concept, the channel is given with the relative global relevance score (with respect to channel 102 in features.28) in parentheses. Following the relevance flow, concept ‘animal on branch’ is dependent on concepts describing the branch (for example, ‘wood (horizontal)’ and ‘brown, knobby’) and colourful plumage (for example, ‘colourful feathers’ and ‘colourful threads’). Additional examples can be found in Supplementary Note 5. Credit: iStock.com/Thomas Marx, iStock.com/erniedecker.
