Extended Data Fig. 3: Loss of plasticity in Online Permuted MNIST.

a, Left, an MNIST image with the label ‘7’; right, a corresponding permuted image. b, Loss of plasticity in Online Permuted MNIST is robust over step sizes, network sizes and rates of change. c, Evolution of various qualities of a deep network trained by means of backpropagation with different step sizes. Left, over time, the percentage of dead units in the network increases. Centre, the average magnitude of the weights increases over time. Right, the effective rank of the representation of the networks trained with backpropagation decreases over time. The results in these six plots are the average over 30 runs. The solid lines represent the mean and the shaded regions correspond to ±1 standard error. For some lines, the shaded region is thinner than the line width, as standard error is small.