Fig. 3: improv handles concurrent neural activity and behavioral video data streams.

a improv pipeline for processing behavioral video and neural activity traces, implementing streaming dimension reduction, streaming ridge regression, and real-time visualization. b Video frames were streamed from disk at the original data rate of 30 frames/sec. After downsampling images, a ‘proSVD’ actor implemented the dimension reduction algorithm. The learned 10-dimensional proSVD basis vectors stabilized within less than a minute. c The ‘Regression Model’ actor received the dimension-reduced video data and neural activity traces and computed the regression coefficients β. For model fitting, ridge regression was implemented via a streaming update algorithm in which each datum is only seen once. Here, Y represents the matrix of neural data (57 neurons x time) and X represents the matrix of reduced behavioral data (10 proSVD dimensions x time). Different gray lines correspond to different coefficients for each latent behavioral feature (10 total). d Two data visualization methods were used for monitoring during a simulated experiment. Left, Video data (dots) were plotted in the proSVD space with a representative trial highlighted in orange. Right, Regression coefficients were normalized to the top coefficient and overlaid back onto the original behavioral image by projecting from the proSVD basis. Regions of the mouse’s face and paws are most predictive of the simultaneously occurring neural activity (cf., Musall⁴⁹, Fig. 3h).