Fig. 2: improv provides streaming model-based characterization of neural function.

a Diagram showing the conceptual flow of data among all actors in the pipeline. Fluorescence images and visual stimuli data were acquired, preprocessed, fed into the model, and visualized, all in real-time. b Schematic of calcium imaging in zebrafish. An acquisition computer acquired fluorescence images and controlled the projection of visual stimuli, and a second networked computer running improv received data for processing in real time. c The ‘2p Acquisition’ actor was responsible only for sending images from the two-photon microscope to the improv computer, one image at a time (3.6 frames/s). d The ‘Visual Stimuli’ actor broadcast information about the stimulus status and displayed visual stimuli. Stimuli were interleaved moving (4.2 s) and stationary (5.3 s) square wave gratings drifting in eight directions (arrow wheel). e Each image was streamed to the CaImAn Online algorithm, encapsulated in a custom actor, that calculated neural spatial masks (ROIs), fluorescence traces, and estimated (deconvolved) spikes across time for each neuron, shown for three example traces. f A linear-nonlinear-Poisson (LNP) model was reformulated to work in the streaming, one-frame-at-a-time, setting. Center, Diagram of our model incorporating stimuli, self-history, and weighted connection terms. Bottom, Log-likelihood fit over time, as more frames were fed into improv. The online model fit converged to the offline fit obtained using the full data set (dotted line) after a single repetition of unique visual stimuli (shaded region). g The ‘Data Visualization’ actor was a GUI that displayed the raw acquired images (left), a processed image overlaid with ROIs color-coded for directional preference, neural traces for one selected neuron (white), and population average (red), tuning curves, and model metrics. The processed frame showed each neuron colored by its directional selectivity (e.g., green hues indicate forward motion preference). The LNP actor interactively estimated the strongest functional connections to a selected neuron (green lines). The LNP model likelihood function (bottom right) showed the optimization loss across time and estimated connectivity weights of highly connected neurons below.