Fig. 8: Comparison of standard mini-batch optimization and mini-batch optimization with line-search.
From: Mini-batch optimization enables training of ODE models on large-scale datasets
Left panel: Standard mini-batch optimization uses a prescheduled learning rate, which determines the step size during optimization regardless of whether an optimization step leads to an improvement or not. Right panel: If line-search is enabled, the objective function is reevaluated on the same mini-batch and checked for improvement. If no improvement is achieved, the learning rate is reduced until either improvement is achieved or until the maximum number of line-search steps is reached.
