Fig. 8: Manifold property changes by network building blocks.
From: Separability and geometry of object manifolds in deep neural networks
Changes in the relative manifold properties between the input and the output of different network building blocks, shown as change in dimension vs change in center correlations (top) and change in radius vs change in center correlations (bottom). Each panel pools results from a specific building block in AlexNet (blue markers) and VGG-16 (green markers) for both point-cloud manifolds (full class, top 10%) and smooth manifolds (1-d and 2-d, translation and shear). Marker shape represents layer type (square—convolution layer, right-triangle—max-pooling layer, hexagon—fully connected layer). For layer sequences marker shape represents the last layer in the sequence. For isolated ReLU marker shape represent previous layer type; pentagon—ReLU after fully connected layer, up-triangle—ReLU after convolution layer. Color changes from dark to light along the network. a Changes in manifold properties for isolated ReLU operations. b Changes in manifold properties for isolated Max-pooling operations. c Changes in manifold properties for a common sequence of operations: one or more repetitions of convolution, ReLU operations, with or without intermediate normalization operation, ending with a max-pooling operation. d Changes in manifold properties for a common sequence of operations: fully connected, ReLU operations. The data analyzed here correspond to the first set of objects used in the analysis of the mean and confidence interval presented in Figs. 6 and 7, and include additional intermediate values not presented in those plots, notably the inputs of ReLU operations.
