Fig. 2: Movement encoding varies across brain areas.

a, A two-dimensional spatial map of brain-wide prediction of neural activity from video. Each voxel is 150 × 150 µm² in the sagittal plane and spans the brain in the third dimension. Color corresponds to mean variance explained by the embedding-based pipeline over all the neurons contained within each voxel. For visualization a 3 × 3-voxel median filter was applied on the heatmap. Scale bar, 0.5 mm. b, Performance of video-based prediction with neurons pooled according to brain area. Error bars correspond to the s.e.m. of insertion-averaged values. Overlaid markers correspond to individual insertions; numbers of insertions are: medulla n = 36 insertions, midbrain n = 79 insertions, ALM n = 77 insertions, striatum n = 67 insertions, thalamus n = 62. Visual representation of statistics corresponds to pairwise comparison for the embedding-based predictions with two-sided Mann–Whitney U test and Bonferroni correction, ***P < 0.001, NS P > 0.05; for exact P values see Extended Data Fig. 2. c, Comparison of single-neuron explained variance between marker-based method (x axis) and embedding-based method (y axis). Each dot corresponds to a neuron. d, Comparison of single-neuron explained variance between marker-based method (x axis) and end-to-end learning (y axis). Each dot corresponds to a neuron. e, Comparison of single-neuron explained variance between embedding-based method (x axis) and end-to-end learning (y axis). Each dot corresponds to a neuron. Session-averaged improvements in explained variance are, for embedding-based approach versus marker-based approach, mean improvement = 155 ± 5% s.e.m., n = 105 sessions, P < 0.001; end-to-end learning versus marker-based approach, mean improvement = 330 ± 9% s.e.m., n = 105 sessions, P < 0.001; end-to-end learning versus embedding-based approach, mean improvement = 76 ± 3% s.e.m., n = 105 sessions, P < 0.001, one-sided Wilcoxon signed-tank test with Bonferroni correction. f, Spike rates of four example neurons during four single trials. Spike rates are plotted (black line) overlaid with their prediction from the marker-based (green), embedding-based (blue) and end-to-end (brown) pipelines. Note, negative spike rate could have been removed post hoc but here we show raw prediction output. The explained variances for the four neurons are the following: from top left to bottom right: medulla neuron, explained variance marker 0.44, embedding 0.68, end-to-end 0.83; medulla neuron, explained variance marker 0.17, embedding 0.36, end-to-end 0.39; medulla neuron, explained variance marker 0.13, embedding 0.23, end-to-end 0.26; medulla neuron, explained variance marker 0.22, embedding 0.25, end-to-end 0.46.