Fig. 3: Comparison of GLE and AM/BPTT in Fourier space.

a Effect of the individual and combined GLE operators \({{{\mathcal{I}}}}_{\tau }^{-}\) and \({{{\mathcal{D}}}}_{\tau }^{+}\) with shared time constant τ on a single frequency component of an input current I. \({{{\mathcal{I}}}}_{\tau }^{-}\) generates a negative phase shift (towards later times) and sub-unit gain. \({{{\mathcal{D}}}}_{\tau }^{+}\) is its exact inverse and generates a positive phase shift (towards earlier times) and supra-unit gain. b Phase shift and c gain of all four temporal operators in GLE and AM/BPTT across a wide range of the frequency spectrum. Note how the prospective operators \({{{\mathcal{D}}}}_{\tau }^{+}\) and \({{{\mathcal{I}}}}_{\tau }^{+}\) (orange) have the same shift but inverse gain; the same holds for the retrospective operators \({{{\mathcal{I}}}}_{\tau }^{-}\) and \({{{\mathcal{D}}}}_{\tau }^{-}\) (purple). d Phase shift and e gain of the combined operators as they appear in the neuron dynamics. Here, we choose an example forward neuron (blue) with a retrospective attention window (τ^m = 10τ^r). Both the associated GLE errors e (blue) and the adjoint variables λ (dotted) are prospective and precisely invert this phase shift, albeit with a different gain.