Fig. 3: Mode-Assisted training and contrastive divergence performance on the shifting bar dataset.
From: Mode-assisted unsupervised learning of restricted Boltzmann machines
Shown are the Kullback-Liebler (KL) divergences achieved on the binary shifting bar dataset across 25 randomly initialized 14 × 10 restricted Boltzmann machines for both contrastive divergence (CD) with k = 1 iteration and mode-assisted training (MT). In addition, every time a mode sample is taken, CD is allowed to run with k = 720 iterations, a number scaled to the equivalent computational cost of taking a mode sample. Thus, the x-axis can also be read as wall time. The bold line represents the median KL divergence across the runs, and the max/min KL divergences achieved at that training iteration define the shaded area. (a) is with a small CD learning rate, ϵCD = 0.05. (b) is with an exponentially decaying ϵCD(n) = e−cn with decay constant c = 4 and n ∈ [0, 1] being the fraction of completed training iterations.
