Fig. 5: GRAPES applied to FA and DFA on Extended and Fashion MNIST.
From: Introducing principles of synaptic integration in the optimization of deep neural networks
Test accuracy and convergence rate in terms of slowness value for three-hidden layer ReLU networks, with 10% dropout, trained with FA on the Extended MNIST and DFA on the Fashion MNIST dataset, as a function of the layer size and the learning rate. The slowness parameter is computed by fitting the initial 100 epochs. The accuracy for each run is computed as the mean of the test accuracy over the last 10 training epochs. The reported result is the mean over the accuracy of ten independent runs. a Test accuracy and b convergence rate for FA on Extended MNIST. c Test accuracy and d convergence rate for DFA on Fashion MNIST. For visualization purposes, the bases of the SGD and GRAPES bars are slightly shifted from each other. The actual learning rates and layer sizes are the same for both and are reported in the axes' labels.
