Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Review Article
  • Published:

Joint modelling of brain and behaviour dynamics with artificial intelligence

Nature Reviews Neuroscience volume 27, pages 87–100 (2026)Cite this article

Subjects

Abstract

Artificial intelligence has created tremendous advances for many scientific and engineering applications. In this Review, we synthesize recent advances in joint brain–behaviour modelling of neural and behavioural data, with a focus on methodological innovations, scientific and technical motivations, and key areas for future innovation. We discuss how these tools reveal the shared structure between the brain and behaviour and how they can be used for both science and engineering aims. We highlight how three broad classes with differing aims — discriminative, generative and contrastive — are shaping joint modelling approaches. We also discuss recent advances in behavioural analysis approaches, including pose estimation, hierarchical behaviour analysis and multimodal-language models, which could influence the next generation of joint models. Finally, we argue that considering not only the performance of models but also their trustworthiness and interpretability metrics can help to advance the development of joint modelling approaches.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on SpringerLink
  • Instant access to the full article PDF.

USD 39.95

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Common neural network architectures.
Fig. 2: Three broad classes of neural–behavioural dynamics models.
Fig. 3: Hierarchical behavioural analysis.

Similar content being viewed by others

References

  1. Krakauer, J. W., Ghazanfar, A. A., Gomez-Marin, A., MacIver, M. A. & Poeppel, D. Neuroscience needs behavior: correcting a reductionist bias. Neuron 93, 480–490 (2017).

    Article  CAS  PubMed  Google Scholar 

  2. Pereira, T. D., Shaevitz, J. W. & Murthy, M. Quantifying behavior to understand the brain. Nat. Neurosci. 23, 1537–1549 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  3. Mathis, M. W., Rotondo, A. P., Chang, E. F., Tolias, A. S. & Mathis, A. Decoding the brain: from neural representations to mechanistic models. Cell 187, 5814–5832 (2024).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  4. Siegle, J. H. et al. Open Ephys: an open-source, plugin-based platform for multichannel electrophysiology. J. Neural Eng. 14, 045003 (2017).

    Article  PubMed  Google Scholar 

  5. Siegle, J. H. et al. Survey of spiking in the mouse visual system reveals functional hierarchy. Nature 592, 86–92 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  6. Helmchen, F. & Denk, W. Deep tissue two-photon microscopy. Nat. Methods 2, 932–940 (2005).

    Article  CAS  PubMed  Google Scholar 

  7. Mathis, M. W. & Mathis, A. Deep learning tools for the measurement of animal behavior in neuroscience. Curr. Opin. Neurobiol. 60, 1–11 (2020).

    Article  CAS  PubMed  Google Scholar 

  8. Hong, G. & Lieber, C. M. Novel electrode technologies for neural recordings. Nat. Rev. Neurosci. 20, 330–345 (2019).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  9. Manley, J. et al. Simultaneous, cortex-wide dynamics of up to 1 million neurons reveal unbounded scaling of dimensionality with neuron number. Neuron 112, 1694–1709.e5 (2024).

    Article  PubMed  PubMed Central  Google Scholar 

  10. Anderson, D. J. & Perona, P. Toward a science of computational ethology. Neuron 84, 18–31 (2014).

    Article  CAS  PubMed  Google Scholar 

  11. von Ziegler, L., Sturman, O. & Bohacek, J. Big behavior: challenges and opportunities in a new era of deep behavior profiling. Neuropsychopharmacology 46, 33–44 (2021).

    Article  Google Scholar 

  12. Tuia, D. et al. Perspectives in machine learning for wildlife conservation. Nat. Commun. 13, 1–15 (2022).

    Article  Google Scholar 

  13. Couzin, I. D. & Heins, C. Emerging technologies for behavioral research in changing environments. Trends Ecol. Evol. 38, 346–354 (2023).

    Article  PubMed  Google Scholar 

  14. Jumper, J. et al. Highly accurate protein structure prediction with alphafold. Nature 596, 583–589 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  15. Andrychowicz, M. et al. Deep learning for day forecasts from sparse observations. Preprint at https://arxiv.org/abs/2306.06079 (2023).

  16. Vyas, S., Golub, M. D., Sussillo, D. & Shenoy, K. V. Computation through neural population dynamics. Annu. Rev. Neurosci. 43, 249–275 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  17. Hurwitz, C. L., Kudryashova, N. N., Onken, A. & Hennig, M. H. Building population models for large-scale neural recordings: opportunities and pitfalls. Curr. Opin. Neurobiol. 70, 64–73 (2021).

    Article  CAS  PubMed  Google Scholar 

  18. Wöhr, M. & Schwarting, R. K. Affective communication in rodents: ultrasonic vocalizations as a tool for research on emotion and motivation. Cell Tissue Res. 354, 81–97 (2013).

    Article  PubMed  Google Scholar 

  19. Ishiyama, S. & Brecht, M. Neural correlates of ticklishness in the rat somatosensory cortex. Science 354, 757–760 (2016).

    Article  CAS  PubMed  Google Scholar 

  20. Hassabis, D., Kumaran, D., Summerfield, C. & Botvinick, M. Neuroscience-inspired artificial intelligence. Neuron 95, 245–258 (2017).

    Article  CAS  PubMed  Google Scholar 

  21. Mathis, M. W. Adaptive intelligence: leveraging insights from adaptive behavior in animals to build flexible AI systems. Preprint at https://arxiv.org/abs/2411.15234 (2025).

  22. LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

    Article  CAS  PubMed  Google Scholar 

  23. Goodfellow, I., Bengio, Y. & Courville, A. Deep Learning (MIT Press, 2016).

  24. Prince, S. J. D. Understanding Deep Learning (MIT Press, 2023).

    Google Scholar 

  25. Augustine, M. T. A survey on universal approximation theorems. Preprint at https://doi.org/10.48550/arXiv.2407.12895 (2024).

  26. Yamins, D. L. & DiCarlo, J. J. Using goal-driven deep learning models to understand sensory cortex. Nat. Neurosci. 19, 356–365 (2016).

    Article  CAS  PubMed  Google Scholar 

  27. Schölkopf, B. & Smola, A. J. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond (MIT Press, 2002).

  28. Ye, J. & Pandarinath, C. Representation learning for neural population activity with neural data transformers. Neurons Behav. Data Anal. Theory 5, 1–18 (2021).

    Google Scholar 

  29. Azabou, M. et al. A unified, scalable framework for neural population decoding. In Proc. 37th Conf. Neural Inf. Process. Syst. 44937–44956 (Curran Associates, 2023).

  30. Pei, F. et al. Neural latents benchmark ’21: evaluating latent variable models of neural population activity. Preprint at https://arxiv.org/abs/2109.04463 (2022).

  31. Zhou, Z. et al. in Human Brain Artificial Intelligence (eds Liu, Q. at al.) 192–206 (Springer, 2024).

  32. Candelori, B. et al. Spatio-temporal transformers for decoding neural movement control. J. Neural Eng. 22, 016023 (2025).

    Article  Google Scholar 

  33. Metzger, S. L. et al. A high-performance neuroprosthesis for speech decoding and avatar control. Nature 620, 1037–1046 (2023).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  34. Jaegle, A. et al. Perceiver: general perception with iterative attention. In Proc. 38th Int. Conf. Mach. Learn. (eds Meila, M. & Zhang, T.) 4651–4664 (PMLR, 2021).

  35. Vaswani, A. et al. Attention is all you need. In Proc. 31st Int. Conf. Neural Inf. Process. Syst. (eds von Luxburg, U. et al.) 6000–6010 (Curran Associates, 2017).

  36. Havrilla, A. & Liao, W. Understanding scaling laws with statistical and approximation theory for transformer neural networks on intrinsically low-dimensional data. In Proc. 38th Conf. Neural Inf. Process. Syst. (eds Globerson, A. et al.) 42162–42210 (Curran Associates, 2024).

  37. Shaeri, M. et al. A 2.46-mm2 miniaturized brain–machine interface (MiBMI) enabling 31-class brain-to-text decoding. IEEE J. Solid-State Circuits 59, 3566–3579 (2024).

    Article  Google Scholar 

  38. Sani, O. G., Pesaran, B. & Shanechi, M. Dissociative and prioritized modeling of behaviorally relevant neural dynamics using recurrent neural networks. Nat. Neurosci. 27, 2033–2045 (2024).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  39. Kingma, D. P. & Welling, M. Auto-encoding variational bayes. Preprint at https://doi.org/10.48550/arXiv.1312.6114 (2022).

  40. Rezende, D. J., Mohamed, S. & Wierstra, D. Stochastic backpropagation and approximate inference in deep generative models. In Proc. 31st Int. Conf. Mach. Learn. (eds Xing, E. P. & Jebara, T.) 1278–1286 (PMLR, 2014).

  41. Pandarinath, C. et al. Inferring single-trial neural population dynamics using sequential auto-encoders. Nat. Methods 15, 805–815 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  42. Keshtkaran, M. R. et al. A large-scale neural network training framework for generalized estimation of single-trial population dynamics. Nat. Methods 19, 1572–1577 (2022).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  43. Kato, S. et al. Global brain dynamics embed the motor command sequence of caenorhabditis elegans. Cell 163, 656–669 (2015).

    Article  CAS  PubMed  Google Scholar 

  44. Gao, Y., Archer, E., Paninski, L. & Cunningham, J. P. Linear dynamical neural population models through nonlinear embeddings. In Proc. 30th Conf. Neural Inf. Process. Syst. (Curran Associates, 2016).

  45. Hu, A. et al. Modeling latent neural dynamics with gaussian process switching linear dynamical systems. In Proc. 38th Conf. Neural Inf. Process. Syst. (eds Globerson, A. et al.) 33805–33835 (Curran Associates, 2024).

  46. Liu, M., Nair, A., Coria, N., Linderman, S. W. & Anderson, D. J. Encoding of female mating dynamics by a hypothalamic line attractor. Nature 634, 901–909 (2024).

    Article  PubMed  PubMed Central  Google Scholar 

  47. Bird, A., Williams, C. K. & Hawthorne, C. Multi-task dynamical systems. J. Mach. Learn. Res. 23, 1–52 (2022).

    Google Scholar 

  48. Liu, X. et al. Self-supervised learning: generative or contrastive. IEEE Trans. Knowl. Data Eng. 35, 857–876 (2021).

    Google Scholar 

  49. Dhariwal, P. & Nichol, A. Diffusion models beat gans on image synthesis. In Proc. 35th Int. Conf. Neural Inf. Process. Syst. (eds Ranzato, M. et al.) 8780–8794 (Curran Associates, 2021).

  50. Kapoor, J. et al. Latent diffusion for neural spiking data. In Proc. 38th Conf. Neural Inf. Process. Syst. (eds Globerson, A. et al.) 118119–118154 (Curran Associates, 2024).

  51. Hyvärinen, A. & Pajunen, P. Nonlinear independent component analysis: Existence and uniqueness results. Neural Netw. 12, 429–439 (1999).

    Article  PubMed  Google Scholar 

  52. Oord, A. V. d., Li, Y. & Vinyals, O. Representation learning with contrastive predictive coding. Preprint at https://doi.org/10.48550/arXiv.1807.03748 (2019).

  53. Zimmermann, R. S., Sharma, Y., Schneider, S., Bethge, M. & Brendel, W. Contrastive learning inverts the data generating process. In Proc. 38th Int. Conf. Mach. Learn. (eds Meila, M. & Zhang, T.) 12979–12990 (PMLR, 2021).

  54. Schneider, S., Lee, J. H. & Mathis, M. W. Learnable latent embeddings for joint behavioural and neural analysis. Nature 617, 360–368 (2023).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  55. Schneider, S., Laiz, R. G., Filippova, A., Frey, M. & Mathis, M. W. Time-series attribution maps with regularized contrastive learning. In Proc. 28th Int. Conf. Artif. Intell. Stat. (PMLR, 2025).

  56. Chen, T., Kornblith, S., Norouzi, M. & Hinton, G. A simple framework for contrastive learning of visual representations. In Proc. 37th Int. Conf. Mach. Learn. (eds Daumé III, H. & Singh, A.) 1597–1607 (PMLR, 2020).

  57. Laiz, R. G., Schmidt, T. & Schneider, S. Self-supervised contrastive learning performs non-linear system identification. In Proc. 13th Int. Conf. Learn. Represent. (ICLR, 2025).

  58. Gosztolai, A., Peach, R. L., Arnaudon, A., Barahona, M. & Vandergheynst, P. Marble: interpretable representations of neural population dynamics using geometric deep learning. Nat. Methods 22, 612–620 (2025).

    Article  PubMed  PubMed Central  Google Scholar 

  59. Zhou, D. & Wei, X. Learning identifiable and interpretable latent models of high-dimensional neural activity using PI-VAE. In Proc. 35th Int. Conf. Neural Inf. Process. Syst. (Curran Associates, 2020).

  60. Khemakhem, I., Kingma, D. P. & Hyvärinen, A. Variational autoencoders and nonlinear ICA: a unifying framework. In Proc. Int. Conf. Artif. Intell. Stat. (PMLR, 2019).

  61. Lashley, K. S. et al. The Problem of Serial Order in Behavior Vol. 21 (Bobbs-Merrill, 1951).

  62. Tinbergen, N. On aims and methods of ethology. Z. Tierpsychol. 20, 410–433 (1963).

    Article  Google Scholar 

  63. Botvinick, M. M. Hierarchical models of behavior and prefrontal function. Trends Cogn. Sci. 12, 201–208 (2008).

    Article  PubMed  PubMed Central  Google Scholar 

  64. Winter, D.Biomechanics and Motor Control of Human Movement (Wiley, 2009).

  65. Girshick, R., Donahue, J., Darrell, T. & Malik, J. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proc. IEEE Conf. Comput. Vision Pattern Recogn. 580–587 (IEEE, 2014).

  66. Kirillov, A. et al. Segment anything. In Proc. IEEE Conf. Comput. Vision 4015–4026 (IEEE, 2023).

  67. Ravi, N. et al. SAM 2: segment anything in images and videos. In Proc. 13th Int. Conf. Learn. Represent. (ICLR, 2025).

  68. Romero-Ferrero, F., Bergomi, M. G., Hinz, R. C., Heras, F. J. & de Polavieja, G. G. idtracker. ai: tracking all individuals in small or large collectives of unmarked animals. Nat. Methods 16, 179 (2019).

    Article  CAS  PubMed  Google Scholar 

  69. Walter, T. & Couzin, I. D. TRex, a fast multi-animal tracking system with markerless identification, and 2D estimation of posture and visual fields. eLife 10, e64000 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  70. Mathis, A., Schneider, S., Lauer, J. & Mathis, M. W. A primer on motion capture with deep learning: principles, pitfalls, and perspectives. Neuron 108, 44–65 (2020).

    Article  CAS  PubMed  Google Scholar 

  71. Mathis, A. et al. DeepLabCut: markerless pose estimation of user-defined body parts with deep learning. Nat. Neurosci. 21, 1281 (2018).

    Article  CAS  PubMed  Google Scholar 

  72. Nath, T. et al. Using DeepLabCut for 3D markerless pose estimation across species and behaviors. Nat. Protoc. 14, 2152–2176 (2019).

    Article  CAS  PubMed  Google Scholar 

  73. Segalin, C. et al. The Mouse Action Recognition System (MARS) software pipeline for automated analysis of social behaviors in mice. eLife 10, e63720 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  74. Lauer, J. et al. Multi-animal pose estimation, identification and tracking with deeplabcut. Nat. Methods 19, 496–504 (2022).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  75. Ye, S. et al. Superanimal pretrained pose estimation models for behavioral analysis. Nat. Commun. 15, 5165 (2024).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  76. Pereira, T. D. et al. SLEAP: a deep learning system for multi-animal pose tracking. Nat. Methods 19, 486–495 (2022).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  77. Zhou, M., Stoffl, L., Mathis, M. W. & Mathis, A. Rethinking pose estimation in crowds: overcoming the detection information bottleneck and ambiguity. In Proc. IEEE Conf. Comput. Vision 14689–14699 (IEEE, 2023).

  78. Beery, S., Morris, D. & Yang, S. Efficient pipeline for camera trap image review. Preprint at https://doi.org/10.48550/arXiv.1907.06772 (2019).

  79. Dunn, T. W. et al. Geometric deep learning enables 3D kinematic profiling across species and environments. Nat. Methods 18, 564–573 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  80. Karashchuk, P. et al. Anipose: a toolkit for robust markerless 3D pose estimation. Cell Rep. 36, 109730 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  81. Joska, D. et al. AcinoSet: a 3D pose estimation dataset and baseline models for cheetahs in the wild. In 2021 IEEE Int. Conf. Robot. Autom. (ICRA) 13901–13908 (IEEE, 2021).

  82. Kaneko, T. et al. Deciphering social traits and pathophysiological conditions from natural behaviors in common marmosets. Curr. Biol. 34, 2854–2867 (2024).

    Article  CAS  PubMed  Google Scholar 

  83. Yurimoto, T. et al. Development of a 3D tracking system for multiple marmosets under free-moving conditions. Commun. Biol. 7, 216 (2024).

    Article  PubMed  PubMed Central  Google Scholar 

  84. Wiltschko, A. B. et al. Mapping sub-second structure in mouse behavior. Neuron 88, 1121–1135 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  85. Weinreb, C. et al. Keypoint-MoSeq: parsing behavior by linking point tracking to pose dynamics. Nat. Methods 21, 1329–1339 (2024).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  86. Menegas, W. et al. High-throughput unsupervised quantification of patterns in the natural behavior of marmosets. eLife 13, RP103586 (2024).

    Google Scholar 

  87. Gosztolai, A. et al. LiftPose3D, a deep learning-based approach for transforming two-dimensional to three-dimensional poses in laboratory animals. Nat. Methods 18, 975–981 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  88. Hu, B. et al. 3D mouse pose from single-view video and a new dataset. Sci. Rep. 13, 13554 (2023).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  89. DeWolf, T., Schneider, S., Soubiran, P., Roggenbach, A. & Mathis, M. W. Neuro-musculoskeletal modeling reveals muscle-level neural dynamics of adaptive learning in sensorimotor cortex. Preprint at bioRxiv https://doi.org/10.1101/2024.09.11.612513 (2024).

  90. Hampali, S., Sarkar, S. D., Rad, M. & Lepetit, V. Keypoint transformer: solving joint identification in challenging hands and object interactions for accurate 3D pose estimation. In Proc. IEEE/CVF Conf. Comput. Vision Pattern Recogn. (CVPR) 11090–11100 (IEEE, 2022).

  91. Qi, H., Zhao, C., Salzmann, M. & Mathis, A. HOISDF: constraining 3D hand–object pose estimation with global signed distance fields. In Proc. IEEE/CVF Conf. Comput. Vision Pattern Recogn. (CVPR) 10392–10402 (IEEE, 2024).

  92. de Chaumont, F. et al. Real-time analysis of the behaviour of groups of mice via a depth-sensing camera and machine learning. Nat. Biomed. Eng. 3, 930–942 (2019).

    Article  PubMed  Google Scholar 

  93. Ye, S., Lauer, J., Zhou, M., Mathis, A. & Mathis, M. AmadeusGPT: a natural language interface for interactive animal behavioral analysis. In Proc. 37th Int. Conf. Neural Inf. Process. Syst. 6297-6329 (Curran Associates, 2023).

  94. Ding, G., Sener, F. & Yao, A. Temporal action segmentation: an analysis of modern techniques. IEEE Trans. Pattern Anal Mach. Intell. 46, 1011–1030 (2023).

    Article  Google Scholar 

  95. Bohnslav, J. P. et al. DeepEthogram, a machine learning pipeline for supervised behavior classification from raw pixels. eLife 10, e63377 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  96. Camilleri, M. P., Bains, R. S. & Williams, C. K. Of mice and mates: automated classification and modelling of mouse behaviour in groups using a single model across cages. Int. J. Comput. Vis. 132, 5491–5513 (2024).

    Article  PubMed  PubMed Central  Google Scholar 

  97. Kabra, M., Robie, A. A., Rivera-Alba, M., Branson, S. & Branson, K. JAABA: interactive machine learning for automatic annotation of animal behavior. Nat. Methods 10, 64 (2013).

    Article  CAS  PubMed  Google Scholar 

  98. Sturman, O. et al. Deep learning-based behavioral analysis reaches human accuracy and is capable of outperforming commercial solutions. Neuropsychopharmacology 45, 1942–1952 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  99. Sun, J. J. et al. MABe22: a multi-species multi-task benchmark for learned representations of behavior. In Proc. 40th Int. Conf. Mach. Learn. (eds Krause, A. et al.) 32936–32990 (PMLR, 2023).

  100. Bordes, J. et al. Automatically annotated motion tracking identifies a distinct social behavioral profile following chronic social defeat stress. Nat. Commun. 14, 4319 (2023).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  101. Goodwin, N. L. et al. Simple behavioral analysis (SimBA) as a platform for explainable machine learning in behavioral neuroscience. Nat. Neurosci. 27, 1411–1424 (2024).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  102. Kozlova, E., Bonnetto, A. & Mathis, A. DLC2Action: a deep learning-based toolbox for automated behavior segmentation. Preprint at bioRxiv https://doi.org/10.1101/2025.09.27.678941 (2025).

  103. Madan, N., Moegelmose, A., Modi, R., Rawat, Y. S. & Moeslund, T. B. Foundation models for video understanding: a survey. Preprint at arXiv https://doi.org/10.48550/arXiv.2405.03770 (2024).

  104. Feichtenhofer, C., Fan, H., Malik, J. & He, K. Slowfast networks for video recognition. In Proc. EEE/CVF Conf. Comput. Vision Pattern Recogn. (CVPR) 6202–6211 (2019).

  105. Zhu, W. et al. MotionBERT: a unified perspective on learning human motion representations. In Proc. EEE/CVF Conf. Comput. Vision Pattern Recogn. (CVPR) 15085–15099 (2023).

  106. He, K. et al. Masked autoencoders are scalable vision learners. In Proc. EEE/CVF Conf. Comput. Vision Pattern Recogn. (CVPR) 16000–16009 (2022).

  107. Feichtenhofer, C. et al. Masked autoencoders as spatiotemporal learners. Adv. Neural Inf. Process. Syst. 35, 35946–35958 (2022).

    Google Scholar 

  108. Tong, Z., Song, Y., Wang, J. & Wang, L. VideoMAE: masked autoencoders are data-efficient learners for self-supervised video pre-training. In Proc. 36th Int. Conf. Neural Inf. Process. Syst. (Curran Associates, 2022).

  109. Berman, G. J., Choi, D. M., Bialek, W. & Shaevitz, J. W. Mapping the stereotyped behaviour of freely moving fruit flies. J. R. Soc. Interface. 11, 20140672 (2014).

    Article  PubMed  PubMed Central  Google Scholar 

  110. Markowitz, J. E. et al. The striatum organizes 3D behavior via moment-to-moment action selection. Cell 174, 44–58 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  111. Hsu, A. I. & Yttri, E. A. B-SOiD, an open-source unsupervised algorithm for identification and fast prediction of behaviors. Nat. Commun. 12, 5188 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  112. Luxem, K. et al. Identifying behavioral structure from deep variational embeddings of animal motion. Commun. Biol. 5, 1267 (2022).

    Article  PubMed  PubMed Central  Google Scholar 

  113. Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: pre-training of deep bidirectional transformers for language understanding. In Proc. 2019 Conf. North Am. Chapter Assoc. Comput. Linguist. 4171–4186 (ACL, 2019).

  114. Yue, Z. et al. TS2Vec: towards universal representation of time series. In Proc. AAAI Conf. Artif. Intel. 8980–8987 (2022).

  115. Qi, C. R., Su, H., Mo, K. & Guibas, L. J. PointNet: deep learning on point sets for 3D classification and segmentation. In Proc. IEEE Conf. Comput. Vision Pattern Recogn. 652–660 (2017).

  116. Stoffl, L., Bonnetto, A., d’Ascoli, S. & Mathis, A. Elucidating the hierarchical nature of behavior with masked autoencoders. In European Conf. Comput. Vision 106–125 (Springer, 2024).

  117. Azabou, M. et al. Relax, it doesn’t matter how you get there: a new self-supervised approach for multi-timescale behavior analysis. In Proc. 37th Int. Conf. Neural Inf. Process. Syst. 28491–28509 (Curran Associates, 2023).

  118. Tuyttens, F. et al. Observer bias in animal behaviour research: can we believe what we score, if we score what we believe? Anim. Behav. 90, 273–280 (2014).

    Article  Google Scholar 

  119. De Melo, C. M. et al. Next-generation deep learning based on simulators and synthetic data. Trends Cogn. Sci. 26, 174–187 (2022).

    Article  PubMed  Google Scholar 

  120. Bai, J. et al. Qwen-VL: a versatile vision-language model for understanding, localization, text reading, and beyond. Preprint at https://arxiv.org/abs/2308.12966 (2023).

  121. Li, J., Li, D., Xiong, C. & Hoi, S. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Proc. Int. Conf. Mach. Learn. 12888–12900 (PMLR, 2022).

  122. Bai, J. et al. Qwen technical report. Preprint at https://arxiv.org/abs/2309.16609 (2023).

  123. Li, F. et al. LLaVA-NeXT-Interleave: tackling multi-image, video, and 3D in large multimodal models. Preprint at arXiv https://doi.org/10.48550/arXiv.2407.07895 (2024).

  124. Li, B. et al. LLaVA-OneVision: easy visual task transfer. Preprint at https://arxiv.org/abs/2408.03326 (2024).

  125. Ye, S., Qi, H., Mathis, A. & Mathis, M. W. LLaVAction: evaluating and training multi-modal large language models for action recognition. Preprint at https://arxiv.org/abs/2503.18712 (2025).

  126. Vargas, A. M. et al. Task-driven neural network models predict neural dynamics of proprioception. Cell 187, 1745–1761 (2024).

    Article  Google Scholar 

  127. Melis, J. M., Siwanowicz, I. & Dickinson, M. H. Machine learning reveals the control mechanics of an insect wing hinge. Nature 628, 795–803 (2024).

    Article  CAS  PubMed  Google Scholar 

  128. Vaxenburg, R. et al. Whole-body physics simulation of fruit fly locomotion. Nature 643, 1312–1320 (2025).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  129. Buchanan, T. S., Lloyd, D. G., Manal, K. & Besier, T. F. Neuromusculoskeletal modeling: estimation of muscle forces and joint moments and movements from measurements of neural command. J. Appl. Biomech. 20, 367–395 (2004).

    Article  PubMed  PubMed Central  Google Scholar 

  130. Doerig, A. et al. The neuroconnectionist research programme. Nat. Rev. Neurosci. 24, 431–450 (2023).

    Article  CAS  PubMed  Google Scholar 

  131. Wang, E. Y. et al. Foundation model of neural activity predicts response to new stimulus types. Nature 640, 470–477 (2025).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  132. Lipton, Z. C. The mythos of model interpretability: in machine learning, the concept of interpretability is both important and slippery. Queue 16, 31–57 (2018).

    Article  Google Scholar 

  133. Goodfellow, I. J., Shlens, J. & Szegedy, C. Explaining and harnessing adversarial examples. Preprint at arXiv https://doi.org/10.48550/arXiv.1412.6572 (2014).

  134. Ribeiro, M. T., Singh, S. & Guestrin, C. “Why should I trust you?” Explaining the predictions of any classifier. In Proc. 22nd ACM SIGKDD Int. Conf. Knowl. Discov. Data Mining 1135–1144 (ACM, 2016).

  135. Lundberg, S. M. & Lee, S.-I. A unified approach to interpreting model predictions. In Proc. 31st Int. Conf. Neural Inf. Process. Syst. (Curran Associates, 2017).

  136. Jang, H., Kim, C. & Yang, E. Timing: Temporality-aware integrated gradients for time series explanation. In ICLR 2025 Workshop XAI4Sci. (ICLR, 2025).

  137. Jazayeri, M. & Ostojic, S. Interpreting neural computations by examining intrinsic and embedding dimensionality of neural activity. Curr. Opin. Neurobiol. 70, 113–120 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  138. Abid, A., Zhang, M. J., Bagaria, V. K. & Zou, J. Y. Exploring patterns enriched in a dataset with contrastive principal component analysis. Nat. Commun. 9, 2134 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  139. Merk, T. et al. Invasive neurophysiology and whole brain connectomics for neural decoding in patients with brain implants. Nat. Biomed. Eng. https://doi.org/10.1038/s41551-025-01467-9 (2025).

  140. Raghu, M., Gilmer, J., Yosinski, J. & Sohl-Dickstein, J. SVCCA: singular vector canonical correlation analysis for deep learning dynamics and interpretability. In Proc. 31st Int. Conf. Neural Inf. Process. Syst. (Curran Associates, 2017).

  141. Kriegeskorte, N., Mur, M. & Bandettini, P. A. Representational similarity analysis — connecting the branches of systems neuroscience. Front. Syst. Neurosci. https://doi.org/10.3389/neuro.06.004.2008 (2008).

  142. Kornblith, S., Norouzi, M., Lee, H. & Hinton, G. Similarity of neural network representations revisited. In Proc. Int. Conf. Mach. Learn. 3519–3529 (PMLR, 2019).

  143. Williams, A. H. Equivalence between representational similarity analysis, centered kernel alignment, and canonical correlations analysis. In Proc. UniReps 2nd Edn Workshop Unif. Represent. Neural Models (PMLR, 2024).

  144. Barbosa, J. et al. Quantifying differences in neural population activity with shape metrics. Preprint at bioRxiv https://doi.org/10.1101/2025.01.10.632411 (2025).

  145. Williams, A. H., Kunz, E. M., Kornblith, S. & Linderman, S. W. Generalized shape metrics on neural representations. Adv. Neural Inf. Process. Syst. 34, 4738–4750 (2021).

    PubMed  PubMed Central  Google Scholar 

  146. Schönemann, P. H. A generalized solution of the orthogonal procrustes problem. Psychometrika 31, 1–10 (1966).

    Article  Google Scholar 

  147. Ostrow, M., Eisen, A., Kozachkov, L. & Fiete, I. Beyond geometry: comparing the temporal structure of computation in neural circuits with dynamical similarity analysis. In Proc. 37th Int. Conf. Neural Inf. Process. Syst. 33824–33837 (Curran Associates, 2023).

  148. Vidal, M., Wolf, N., Rosenberg, B., Harris, B. P. & Mathis, A. Perspectives on individual animal identification from biology and computer vision. Integr. Comp. Biol. 61, 900–916 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  149. Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Schölkopf, B. & Bachem, O. Challenging common assumptions in the unsupervised learning of disentangled representations. In Proc. Int. Conf. Mach. Learn. 4114–4124 (PMLR, 2019).

  150. Radford, A. & Narasimhan, K. Improving language understanding by generative pre-training. Preprint at https://arxiv.org/abs/1801.06146 (2018).

  151. OpenAI et al. GPT-4 technical report. Preprint at https://doi.org/10.48550/arXiv.2303.08774 (2024).

  152. Radford, A. et al. Learning transferable visual models from natural language supervision. In Proc. Int. Conf. Mach. Learn. 8748–8763 (PMLR, 2021).

  153. Hyvärinen, A., Sasaki, H. & Turner, R. E. Nonlinear ICA using auxiliary variables and generalized contrastive learning. In 22nd Int. Conf. Artif. Intell. Stat. 859–868 (PMLR, 2019).

  154. Brown, T. et al. Language models are few-shot learners. Adv. Neural Inf. Process. Syst. 33, 1877–1901 (2020).

    Google Scholar 

  155. Bommasani, R. et al. On the opportunities and risks of foundation models. Preprint at https://arxiv.org/abs/2108.07258 (2021).

  156. Castro, P. S. et al. Discovering symbolic cognitive models from human and animal behavior. Preprint at bioRxiv https://doi.org/10.1101/2025.02.05.636732 (2025).

Download references

Acknowledgements

The authors thank members of their laboratories, especially M. Simos, P. Muratore and H. Mirzaeri for discussions. This work was funded by the Swiss National Science Foundation (SNSF) though grants 310030_212516 (to A.M.), TMSGI3_226525 (to M.W.M.) and 320030-227871 (to A.M. and M.W.M.).

Author information

Authors and Affiliations

  1. Brain Mind Institute, School of Life Sciences, École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland

    Mackenzie Weygandt Mathis & Alexander Mathis

Authors
  1. Mackenzie Weygandt Mathis
    View author publications

    Search author on:PubMed Google Scholar

  2. Alexander Mathis
    View author publications

    Search author on:PubMed Google Scholar

Contributions

The authors contributed equally to all aspects of the article. A.M. led the behavioural modelling sections, and M.W.M. led the neural and joint modelling sections.

Corresponding author

Correspondence to Mackenzie Weygandt Mathis.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Reviews Neuroscience thanks Michael Yartsev, who co-reviewed with Adam Lowet, and the other, anonymous, reviewer for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Related links

European Charter for the Responsible Development of Neurotechnologies: https://www.braincouncil.eu/european-charter-for-the-responsible-development-of-neurotechnologies/

Glossary

Agent-based systems

Artificial intelligence (AI) systems capable of autonomous goal-directed behaviour, including planning, reasoning and interaction with their environment to achieve specified objectives.

Attribution methods

Techniques that identify which input features contribute most to a model’s output.

Decoder

A network module that transforms latent representations back into the data domain, reconstructing or generating outputs.

Deep learning

A subset of machine learning using multilayer neural networks to learn complex, hierarchical data representations.

Digital twin

A computational replica of a real system used for simulation, prediction or control.

Discrete state transitions

Changes between distinct system states, often modelled as jumps in state-space dynamics.

Embeddings

Vector representations capturing semantic or structural relationships among data elements.

Encoder

A network module that maps inputs into a latent space.

Latent space

The abstract representation space where encoded data are organized by learned features. Latent representations live in the latent space, just as integers live in the set of integers \({\mathbb{Z}}\). It is a space, because it also has structure. For instance, often you can add two latent representations, or take the average.

Machine learning

Algorithms that learn patterns from data to make predictions or decisions without explicit programming.

Neural dynamics

The time-evolving activity patterns and interactions among neurons or artificial network units.

Poisson loss

A likelihood-based loss for count data assuming Poisson-distributed observations. Commonly used to model spike counts in neuroscience. 

Poisson noise

Random variability in count data arising from discrete stochastic events.

Self-supervised learning

Learning representations from unlabelled data such as by predicting masked parts of the input from other parts, or learning from temporal structure.

Supervised learning

Learning from labelled data pairs (x, y) to map inputs x to known outputs y.

Topological data analysis

Method using topology to characterize the shape and structure of complex data.

Universal approximators

Given enough capacity, neural networks can approximate any continuous function on compact domains to arbitrary precision. For example, even a feedforward network with a single hidden layer of sufficient width is a universal approximator.

Zero-shot performance

The performance of a model when evaluated on tasks or samples without training data (from this task/setting). This evaluates generalization. Few-shot evaluation allows training on a few samples.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mathis, M.W., Mathis, A. Joint modelling of brain and behaviour dynamics with artificial intelligence. Nat. Rev. Neurosci. 27, 87–100 (2026). https://doi.org/10.1038/s41583-025-00996-1

Download citation

  • Accepted:

  • Published:

  • Version of record:

  • Issue date:

  • DOI: https://doi.org/10.1038/s41583-025-00996-1

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing