Fig. 4: Real-time latent neural trajectory prediction with improv.

a improv pipeline for dimension-reducing multichannel neural electrophysiology data and predicting latent dynamics in real-time. Data from (O’Doherty et al.⁵³). b Neural spiking data are streamed from disk, simulating online waveform template matching, binned at 10 ms, and smoothed using a Gaussian kernel to obtain firing rates. The ‘proSVD’ actor then reduced 182 units down to a stable 6-dimensional space. c The ‘Bubblewrap’ actor incorporates dimension-reduced neural trajectories and fits (via a streaming EM algorithm) a Gaussian mixture Hidden Markov Model to coarsely tile the neural space. Left, A dimension-reduced input data trajectory (orange line), bubbles (shaded blue ellipses), and the (probabilistic) connections between bubbles (dashed black line). Right, The model predictive performance is quantified by the log predictive probability (blue, top) and the entropy of the learned transition matrix (purple, bottom). Black lines are exponentially weighted moving averages. d Predictions can be qualitatively and quantitatively monitored via improv. Left, Dimension-reduced neural data are displayed in light gray with the neural trajectory of the current arm reach shown in orange; bubbles and connections as in c. The dashed black line indicates the predicted transitions in the space given the first 150 ms of the trial, predicting 400 ms into the future. Right, Bubblewrap’s predictive performance (log predictive probability and entropy; mean and standard deviation) is shown as a function of seconds predicted ahead. Error bars denoting standard deviation are calculated across all timepoints in the second half of the dataset (n = 7500).