Table 1 Computational complexity of models

From: Closed-form continuous-time neural networks

Time complexity

Sequence and time-step prediction complexity

Method

Complexity

Local error

Model

Sequence prediction

Time-step prediction

pth-order solver

\({{{\mathcal{O}}}}(Kp)\)

\({{{\mathcal{O}}}}({\epsilon }^{p+1})\)

RNN

\({{{\mathcal{O}}}}(nk)\)

\({{{\mathcal{O}}}}(k)\)

Adaptive-step solver

—

\({{{\mathcal{O}}}}({\tilde{\epsilon }}^{p+1})\)

ODE-RNN

\({{{\mathcal{O}}}}(nkp)\)

\({{{\mathcal{O}}}}(kp)\)

Euler hypersolver

\({{{\mathcal{O}}}}(K)\)

\({{{\mathcal{O}}}}(\delta {\epsilon }^{2})\)

Transformer

\({{{\mathcal{O}}}}({n}^{2}k)\)

\({{{\mathcal{O}}}}(nk)\)

pth-order hypersolver

\({{{\mathcal{O}}}}(Kp)\)

\({{{\mathcal{O}}}}(\delta {\epsilon }^{p+1})\)

CfC

\({{{\mathcal{O}}}}(nk)\)

\({{{\mathcal{O}}}}(k)\)

CfC (current work)

\({{{\mathcal{O}}}}(\tilde{K})\)

Not relevant

 
  1. Left: The time complexity of the process to compute K solver steps. ϵ is step size. \(\tilde{\epsilon }\) is the maximum step size and δ ≪ 0. \(\tilde{K}\) is the time steps for CfCs corresponding to the input time step, which is typically one to three orders of magnitude smaller than K. The left portion is reproduced with permission from ref. 17. Right: Sequence and time-step prediction complexity. n is the sequence length. k is the number of hidden units. p is the order of the ODE solver.
Back to article page