Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Comment
  • Published:

Interpretability of artificial neural network models in artificial intelligence versus neuroscience

Nature Machine Intelligence volume 4, pages 1065–1067 (2022)Cite this article

Subjects

The notion of ‘interpretability’ of artificial neural networks (ANNs) is of growing importance in neuroscience and artificial intelligence (AI). But interpretability means different things to neuroscientists as opposed to AI researchers. In this article, we discuss the potential synergies and tensions between these two communities in interpreting ANNs.

Access through your institution
Buy or subscribe

This is a preview of subscription content, access via your institution

Relevant articles

Open Access articles citing this article.

Access options

Access through your institution

Buy this article

  • Purchase on SpringerLink
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Interpretability of models for AI and neuroscience.

References

  1. Yamins, D. L. K. et al. Proc. Natl Acad. Sci. USA 111, 8619–8624 (2014).

    Article  Google Scholar 

  2. Schrimpf, M. et al. Proc. Natl Acad. Sci. USA 118, e2105646118 (2021).

    Article  Google Scholar 

  3. Pospisil, D. A., Pasupathy, A. & Bair, W. Elife 7, e38242 (2018).

    Article  Google Scholar 

  4. Bao, P., She, L., McGill, M. & Tsao, D. Y. Nature 583, 103–108 (2020).

    Article  Google Scholar 

  5. Schrimpf, M. et al. Preprint at bioRxiv http://biorxiv.org/lookup/doi/10.1101/407007 (2018).

  6. Kar, K., Kubilius, J., Schmidt, K., Issa, E. B. & DiCarlo, J. J. Nat. Neurosci. 22, 974–983 (2019).

    Article  Google Scholar 

  7. Kar, K. & DiCarlo, J. J. Neuron 109, 164–176 (2021).

    Article  Google Scholar 

  8. European Parliament. Directorate General for Parliamentary Research Services. A governance framework for algorithmic accountability and transparency. (Publications Office, 2019).

  9. Mordvintsev, A., Olah, C., & Tyka, M. Inceptionism: Going deeper into neural networks (2015); https://ai.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html

  10. Majaj, N. J., Hong, H., Solomon, E. A. & DiCarlo, J. J. J. Neurosci. 35, 13402–13418 (2015).

    Article  Google Scholar 

  11. Willeke, K. F. et al. Preprint at arXiv https://doi.org/10.48550/arXiv.2206.08666 (2022).

  12. Conwell, C. et al. SVRHM 2021 Workshop (NeurIPS, 2021).

  13. Holzinger, A. in 2018 World Symposium on Digital Intelligence for Systems and Machines (DISA) 55–66 (IEEE, 2018).

  14. Hooker, S., Erhan, D., Kindermans, P.-J. & Kim, B. Advances in Neural Information Processing Systems 32 (2019).

  15. Gosselin, F. & Schyns, P. G. Vision Res. 41, 2261–2271 (2001).

    Article  Google Scholar 

  16. Murray, R. F. J. Vis. 11, 2 (2011).

    Article  Google Scholar 

  17. Bashivan, P., Kar, K. & DiCarlo, J. J. Science 364, eaav9436 (2019).

    Article  Google Scholar 

  18. Ponce, C. R. et al. Cell 177, 999–1009 (2019).

    Article  Google Scholar 

  19. Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. In International conference on computer vision 618–626 (IEEE, 2017).

  20. Geirhos, R. et al. Advances in Neural Information Processing Systems 34, 23885–23899 (2021).

  21. Zipser, D. & Andersen, R. A. Nature 331, 679–684 (1988).

    Article  Google Scholar 

  22. Mante, V., Sussillo, D., Shenoy, K. V. & Newsome, W. T. Nature 503, 78–84 (2013).

    Article  Google Scholar 

  23. Olshausen, B. A. & Field, D. J. Nature 381, 607–609 (1996).

    Article  Google Scholar 

  24. Olah, C., Mordvintsev, A., Schubert, L. Feature Visualization (Distill, 2017); https://distill.pub/2017/feature-visualization

Download references

Acknowledgements

The authors would like to thank C. Shain for helpful comments and discussions. E.F. was supported by NIH awards R01-DC016607, R01-DC016950 and U01-NS121471, and by research funds from the McGovern Institute for Brain Research, the Brain and Cognitive Sciences Department and the Simons Center for the Social Brain. K.K. was supported by the Canada Research Chair Program. This research was undertaken thanks in part to funding from the Canada First Research Excellence Fund. K.K. was supported by an unrestricted research fund from Google LLC.

Author information

Authors and Affiliations

  1. Department of Biology, Centre for Vision Research, York University, Toronto, Ontario, Canada

    Kohitij Kar

  2. McGovern Institute for Brain Research and Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA

    Kohitij Kar & Evelina Fedorenko

  3. Center for Brains, Minds, and Machines, Massachusetts Institute of Technology, Cambridge, MA, USA

    Kohitij Kar

  4. Brain Team, Google Research, Toronto, Ontario, Canada

    Simon Kornblith

Authors
  1. Kohitij Kar
    View author publications

    Search author on:PubMed Google Scholar

  2. Simon Kornblith
    View author publications

    Search author on:PubMed Google Scholar

  3. Evelina Fedorenko
    View author publications

    Search author on:PubMed Google Scholar

Corresponding author

Correspondence to Kohitij Kar.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Machine Intelligence thanks the anonymous reviewers for their contribution to the peer review of this work.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kar, K., Kornblith, S. & Fedorenko, E. Interpretability of artificial neural network models in artificial intelligence versus neuroscience. Nat Mach Intell 4, 1065–1067 (2022). https://doi.org/10.1038/s42256-022-00592-3

Download citation

  • Published:

  • Issue date:

  • DOI: https://doi.org/10.1038/s42256-022-00592-3

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing