Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Flexible modulation of sequence generation in the entorhinal–hippocampal system

Nature Neuroscience volume 24, pages 851–862 (2021)Cite this article

Subjects

Abstract

Exploration, consolidation and planning depend on the generation of sequential state representations. However, these algorithms require disparate forms of sampling dynamics for optimal performance. We theorize how the brain should adapt internally generated sequences for particular cognitive functions and propose a neural mechanism by which this may be accomplished within the entorhinal–hippocampal circuit. Specifically, we demonstrate that the systematic modulation along the medial entorhinal cortex dorsoventral axis of grid population input into the hippocampus facilitates a flexible generative process that can interpolate between qualitatively distinct regimes of sequential hippocampal reactivations. By relating the emergent hippocampal activity patterns drawn from our model to empirical data, we explain and reconcile a diversity of recently observed, but apparently unrelated, phenomena such as generative cycling, diffusive hippocampal reactivations and jumping trajectory events.

Access through your institution
Buy or subscribe

This is a preview of subscription content, access via your institution

Access options

Access through your institution

Buy this article

  • Purchase on SpringerLink
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: In different neurophysiological, behavioral and cognitive states, distinct modes of sequence generation are active in the entorhinal–hippocampal system.
Fig. 2: Spectral modulation of grid cell activity alters the statistical structure of hippocampal sequence generation.
Fig. 3: Simulating the over-representation of remote, motivationally salient, locations.
Fig. 4: Theta sequences generated during a goal-directed foraging experiment exhibited nonlocal transitions in place coding.
Fig. 5: Generative cycling emerges from minimally autocorrelated sequence generation.
Fig. 6: Spatial trajectories encoded in SWRs exhibit distinct stability modulation between wake and sleep phases.
Fig. 7: The performance of cognitive algorithms depends on the sampling regime.
Fig. 8: Dysregulated spectral modulation cause pathological forms of hippocampal reactivations.

Similar content being viewed by others

Data availability

No data was acquired for this study.

Code availability

Simulation code was written using open source packages in the Python 3 programming environment and can be found at https://github.com/dmcnamee/FlexModEHC.

References

  1. Buzsáki, G. & Moser, E. I. Memory, navigation and theta rhythm in the hippocampal-entorhinal system. Nat. Neurosci. 16, 130–138 (2013).

    Article  PubMed  PubMed Central  Google Scholar 

  2. Rowland, D. C., Roudi, Y., Moser, M.-B. & Moser, E. I. Ten years of grid cells. Annu. Rev. Neurosci. 39, 19–40 (2016).

    Article  CAS  PubMed  Google Scholar 

  3. Lisman, J. et al. Viewpoints: how the hippocampus contributes to memory, navigation and cognition. Nat. Neurosci. 20, 1434–1447 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  4. Ólafsdóttir, H. F., Bush, D. & Barry, C. The role of hippocampal replay in memory and planning. Curr. Biol. 28, R37–R50 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  5. Foster, D. J. Replay comes of age. Annu. Rev. Neurosci. 40, 581–602 (2017).

    Article  CAS  PubMed  Google Scholar 

  6. Klinzing, J. G., Niethard, N. & Born, J. Mechanisms of systems memory consolidation during sleep. Nat. Neurosci. 22, 1598–1610 (2019).

    Article  CAS  PubMed  Google Scholar 

  7. Pfeiffer, B. E. The content of hippocampal ‘replay’. Hippocampus 30, 6–18 (2020).

    Article  PubMed  Google Scholar 

  8. Stella, F., Baracskay, P., O'Neill, J. & Csicsvari, J. Hippocampal reactivation of random trajectories resembling Brownian diffusion. Neuron 102, 450–461 (2019).

    Article  CAS  PubMed  Google Scholar 

  9. Pfeiffer, B. E. & Foster, D. J. Hippocampal place-cell sequences depict future paths to remembered goals. Nature 497, 74–79 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  10. Johnson, A. & Redish, A. Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. J. Neurosci. 27, 12176–12189 (2007).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  11. Wikenheiser, A. M. & Redish, D. Hippocampal theta sequences reflect current goals. Nat. Neurosci. 18, 289–294 (2015).

  12. Kay, K. et al. Constant sub-second cycling between representations of possible futures in the hippocampus. Cell 180, 552–567 (2020).

  13. Buzsáki, G. & Tingley, D. Space and time: the hippocampus as a sequence generator. Trends Cogn. Sci. 22, 853–869 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  14. Káli, S. & Dayan, P. Off-line replay maintains declarative memories in a model of hippocampal-neocortical interactions. Nat. Neurosci. 7, 286–294 (2004).

    Article  PubMed  Google Scholar 

  15. Sanborn, A. & Chater, N. Bayesian brains without probabilities. Trends Cogn. Sci. 20, 883–893 (2016).

    Article  PubMed  Google Scholar 

  16. Matter, M. G. & Daw, N. D. Prioritized memory access explains planning and hippocampal replay.Nat. Neurosci. 21, 1609–1617 (2018).

    Article  Google Scholar 

  17. Sokal, A. in Functional Integration Vol. 361 (eds DeWitt-Morette, C. et al.) 131–192 (Springer, 1997).

  18. McNaughton, B. L. et al. Deciphering the hippocampal polyglot: the hippocampus as a path integration system. J. Exp. Biol. 199, 173–185 (1996).

    Article  CAS  PubMed  Google Scholar 

  19. Burak, Y. & Fiete, I. R. Accurate path integration in continuous attractor network models of grid cells. PLoS Comput. Biol. 5, e1000291 (2009).

    Article  PubMed  PubMed Central  Google Scholar 

  20. Ocko, S., Hardcastle, K., Giocomo, L. & Ganguli, S. Emergent elasticity in the neural code for space. Proc. Natl Acad. Sci. USA 115, E11798–E11806 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  21. Bush, D., Barry, C., Manson, D. & Burgess, N. Using grid cells for navigation. Neuron 87, 507–520 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  22. Behrens, T. et al. What is a cognitive map? Organizing knowledge for flexible behavior. Neuron 100, 490–509 (2018).

    Article  CAS  PubMed  Google Scholar 

  23. Norris, J. Markov Chains (Cambridge Univ. Press, 1997).

  24. Klages, R., Radons, G. & Sokolov, I. M. Anomalous Transport: Foundations and Applications (Wiley, 2008).

  25. McNamee, D. & Wolpert, D. M. Internal models in biological control. Annu. Rev. Control Robot. Auton. Syst. 2, 339–364 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  26. Weber, M. F. & Frey, E. Master equations and the theory of stochastic path integrals. Rep. Prog. Phys. 80, 046601 (2017).

    Article  PubMed  Google Scholar 

  27. McNamee, D., Wolpert, D. & Lengyel, M. Efficient state-space modularization for planning: theory, behavioral and neural signatures. Adv. Neural Inf. Process. Syst. 29, 4511–4519 (2016).

    Google Scholar 

  28. Stachenfeld, K., Botvinick, M. & Gershman, S. The hippocampus as a predictive map. Nat. Neurosci. 20, 1643–1653 (2017).

    Article  CAS  PubMed  Google Scholar 

  29. Michelmann, S., Staresina, B. P., Bowman, H. & Hanslmayr, S. Speed of time-compressed forward replay flexibly changes in human episodic memory. Nat. Hum. Behav. 3, 143–154 (2019).

    Article  PubMed  Google Scholar 

  30. Stensola, H. et al. The entorhinal grid map is discretized. Nature 492, 72–78 (2012).

    Article  CAS  PubMed  Google Scholar 

  31. Barry, C., Ginzberg, L. L., O’Keefe, J. & Burgess, N. Grid cell firing patterns signal environmental novelty by expansion. Proc. Natl Acad. Sci. USA 109, 17687–17692 (2012).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  32. Mallory, C., Hardcastle, K., Bant, J. & Giocomo, L. Grid scale drives the scale and long-term stability of place maps. Nat. Neurosci. 21, 270–282 (2018).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  33. Schlesiger, M. et al. The medial entorhinal cortex is necessary for temporal organization of hippocampal neuronal activity. Nat. Neurosci. 18, 1123–1132 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  34. Yamamoto, J. & Tonegawa, S. Direct medial entorhinal cortex input to hippocampal CA1 is crucial for extended quiet awake replay. Neuron 96, 217–227.e4 (2017).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  35. Viswanathan, G. M. et al. Optimizing the success of random searches. Nature 401, 911–914 (1999).

    Article  CAS  PubMed  Google Scholar 

  36. Pfeiffer, B. E. & Foster, D. J. Autoassociative dynamics in the generation of sequences of hippocampal place cells. Science 349, 180–184 (2015).

    Article  CAS  PubMed  Google Scholar 

  37. Browne, C. B. et al. A survey of Monte Carlo tree search methods. IEEE Trans. Comput. Intell. AI Games 4, 1–43 (2012).

    Article  Google Scholar 

  38. Colgin, L. et al. Frequency of gamma oscillations routes flow of information in the hippocampus. Nature 462, 353–357 (2009).

    Article  CAS  PubMed  Google Scholar 

  39. Dayan, P. Improving generalization for temporal difference learning: the successor representation. Neural Comput. 5, 613–624 (1993).

    Article  Google Scholar 

  40. Suh, J., Foster, D., Davoudi, H., Wilson, M. & Tonegawa, S. Impaired hippocampal ripple-associated replay in a mouse model of schizophrenia. Neuron 80, 484–493 (2013).

    Article  CAS  PubMed  Google Scholar 

  41. Karlsson, M. & Frank, L. Awake replay of remote experiences in the hippocampus. Nat. Neurosci. 12, 913–918 (2009).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  42. Wikenheiser, A. M. & Redish, A. D. Decoding the cognitive map: ensemble hippocampal sequences and decision making. Curr. Opin. Neurobiol. 32, 8–15 (2015).

    Article  CAS  PubMed  Google Scholar 

  43. Gershman, S. J., Horvitz, E. J. & Tenenbaum, J. B. Computational rationality: a converging paradigm for intelligence in brains, minds, and machines. Science 349, 273–278 (2015).

    Article  CAS  PubMed  Google Scholar 

  44. Lieder, F., Griffiths, T. & Hsu, M. Overrepresentation of extreme events in decision making reflects rational use of cognitive resources. Psychological Rev. 125, 1–32 (2018).

    Article  Google Scholar 

  45. Kang, L. & DeWeese, M. Replay as wavefronts and theta sequences as bump oscillations in a grid cell attractor network. eLife 8, e46351 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  46. Witter, M. P., Canto, C. B., Couey, J. J., Koganezawa, N. & O'Reilly, K. C. Architecture of spatial circuits in the hippocampal region. Philos. Trans. R. Soc. B Biol. Sci. 369, 20120515 (2014).

    Article  Google Scholar 

  47. Lisman, J. E. & Jensen, O. The theta-gamma neural code. Neuron 77, 1002–1016 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  48. Hills, T. et al. Exploration versus exploitation in space, mind, and society. Trends Cogn. Sci. 19, 46–54 (2015).

    Article  PubMed  Google Scholar 

  49. Botvinick, M. M., Niv, Y. & Barto, A. C. Hierarchically organized behavior and its neural foundations: a reinforcement learning perspective. Cognition 113, 262–280 (2009).

    Article  PubMed  Google Scholar 

  50. Schapiro, A., Rogers, T., Cordova, N., Turk-Browne, N. & Botvinick, M. Neural representations of events arise from temporal community structure. Nat. Neurosci. 16, 486–492 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  51. Sutton, R. & Barto, A. Reinforcement Learning: An Introduction (MIT Press, 2018).

  52. Cover, T. & Thomas, J. Elements of Information Theory (Wiley, 2006).

  53. Viswanathan, G. M., Da Luz, M. G., Raposo, E. P. & Stanley, H. E. The Physics of Foraging: An Introduction to Random Searches and Biological Encounters (Cambridge Univ. Press, 2011).

  54. Jinnai, Y., Park, J. W., Abel, D. & Konidaris, G. Discovering options for exploration by minimizing cover time. Preprint at arXiv:1903.00606 (2019).

Download references

Acknowledgements

This work was supported by the Center for Brains, Minds and Machines (funded by NSF STC award no. CCF-1231216) and the Wellcome Trust (Sir Henry Wellcome Fellowship grant no. 110257/Z/15/Z to D.C.M.). For the purpose of Open Access, we have applied a CC BY public copyright licence to any Author Accepted Manuscript version arising from this submission. We thank A. Bernacchia, Q. Huys, Z. Kurth-Nelson, M. Nour, E. Russek, H. Sheahan, H. Stojic, S. Tootoonian and E. Wimmer for discussions and feedback. We thank F. Stella and J. Csicsvari (IST Austria) for sharing their data and for performing additional analyses. We thank B. Sorscher and colleagues (Stanford University) for sharing their code.

Author information

Authors and Affiliations

  1. Wellcome Centre for Human Neuroimaging, University College London, London, UK

    Daniel C. McNamee

  2. Max Planck UCL Centre for Computational Psychiatry, London, UK

    Daniel C. McNamee

  3. Department of Psychology, Harvard University, Cambridge, MA, USA

    Daniel C. McNamee & Samuel J. Gershman

  4. Google DeepMind, London, UK

    Kimberly L. Stachenfeld & Matthew M. Botvinick

  5. Gatsby Computational Neuroscience Unit, University College London, London, UK

    Matthew M. Botvinick

  6. Center for Brain Science, Harvard University, Cambridge, MA, USA

    Samuel J. Gershman

  7. Center for Brains, Minds and Machines, MIT, Cambridge, MA, USA

    Samuel J. Gershman

Authors
  1. Daniel C. McNamee
    View author publications

    Search author on:PubMed Google Scholar

  2. Kimberly L. Stachenfeld
    View author publications

    Search author on:PubMed Google Scholar

  3. Matthew M. Botvinick
    View author publications

    Search author on:PubMed Google Scholar

  4. Samuel J. Gershman
    View author publications

    Search author on:PubMed Google Scholar

Contributions

D.C.M., K.L.S., M.M.B. and S.J.G. contributed toward model conceptualization. D.C.M. performed the simulations. D.C.M. prepared a draft of the manuscript. D.C.M., K.L.S., M.M.B. and S.J.G. reviewed the manuscript contributing edits and comments.

Corresponding author

Correspondence to Daniel C. McNamee.

Ethics declarations

Competing interests

D.C.M. and S.J.G. declare no competing interests. K.L.S. and M.M.B. are employed by Google DeepMind.

Additional information

Peer review information Nature Neuroscience thanks Sen Cheng, Kiah Hardcastle and Martin Stemmler for their contribution to the peer review of this work.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1 Sequence generation in an inhomogeneous environment with tempo τ and stability α modulation.

a, Spectral components of the four-room grid world are presented as a function of spatial scale. Generator eigenvectors coincide with the eigenvectors of the corresponding SR. Therefore, many examples of generator eigenvectors may be observed in ref. 28. b, As the tempo parameter τ is reduced (with the stability parameter held at α = 1), the power spectrum sτ,α(λ) reweights the spectral components leading to changes in the spatial scale of the propagation densities ρt. c, The generated sequences vary in the extent of their spatiotemporal traversal of the environment.

Extended Data Fig. 2 Exploration efficiency in a structured environment.

The exploration simulation in Fig. 2 is repeated here but for an environment decomposed into four areas connected by bottlenecks. Each area consisted of 50 × 50 states. Simulation parameters were unchanged from Fig. 2 (Methods). e, Error bands ± s.e.m. for n = 20 trajectories.

Extended Data Fig. 3 Propagator analysis across all possible current and future positions for the circular track task.

a,b, We plot the superdiffusive (panel A, α = 0.5) and diffusive propagators (panel B, α = 1) for the circular track task11 for all current positions (y-axis) with all goals activated. The y-axis (x-axis) indexes the current (future) position. Each row reports the probability of each position being sampled given the current position (the row index). c, The difference in diffusive and superdiffusive propagators is plotted. For example, the row corresponding to the initial X position (marked by the thick black line), shows that the local activations are less likely (red) under superdiffusions compared to diffusion (though they are still possible) while remote goal locations specifically are more likely (blue).

Extended Data Fig. 4 Diffusions are optimized for consolidating directed transition structures.

Three regimes of sequence generation are compared in terms of consolidating a directed policy on the same graph used in Fig. 7 where the policy was assumed to be undirected (that is a random walk). a, The directed policy is dominated by high probability anticlockwise transitions (heavy black arrows) between the clique bottleneck states (highlighted by increased size). Within each clique, transitions to the bottleneck state facilitating an anti-clockwise transition to the next clique are relatively likely to occur. b, The same rank order as in Fig. 7c is observed with the diffusive regime leading to the most accurate consolidation of the directed policy successor representation (SR). c, The true directed policy SR. d–f, The SRs learned via superdiffusion, diffusion, and minimally autocorrelated sampling respectively. Note that the superdiffusive and minimal autocorrelation regimes identify the the dominant anti-clockwise transition structure but fail to reflect the clustering of states based on their clique membership. This simulation indicates that the diffusive regime is optimal for consolidation regardless of whether the target transition structure corresponds to a random walk or not.

Extended Data Fig. 5 Detailed analyses of spectrally modulated propagation in the linear track environment.

a, Copy of Fig. 2d for context and comparative visualization. b, The propagation distributions in panel A are re-expressed as the difference with respect to the baseline (τ = 1, α = 1). c, Each propagation density is plotted against the baseline propagation density (τ = 1, α = 1) on log scales. Note that only superdiffusion results in a non-linear warping of the propagation density. This non-linearity underpins the heavy-tail in the propagation density. d, Small-scale (dashed line) and large-scale (full line) spectral components (that is, generator eigenvectors) are plotted as a function of linear track position. Eigenvalues take a maximum value of 0 and are ordered according to the spatial scale of the corresponding spectral component. For example, eigenvalues close to zero correspond to large-scale spectral components with the spatial scale of the corresponding spectral component decreasing with the eigenvalue which are always less than or equal to zero. Therefore, the scale of a spectral component ϕk decreases as the absolute value |λk| of its corresponding eigenvalue λk increases. e, Spectral power is plotted as a function of eigenvalue on a log scale. As the tempo τ decreases towards zero, the spatiotemporal horizon increases and the propagation density converges on the stationary state distribution. This corresponds to all spectral components with non-zero eigenvalues decaying to zero. Note that diffusion linearly changes the slope of the power spectrum while superdiffusion imposes a non-linear reweighting. This effect underpins the nonlinear time warping in the propagation densities in panel C. f, The characteristic modulation of the power spectrum under diffusion and superdiffusion is demonstrated by the relative power spectrum \(\frac{{s_{\tau ,\alpha }(\lambda )}}{{s_{1,1}(\lambda )}}\) computed as the power ratio with respect to the baseline propagator. Note that the long-range diffusion propagator (α = 1, τ = 0.5, thick red line) downweights small-scale components and upweights large-scale components while superdiffusion (α = 0.5, τ = 1, thick blue line) relatively upweights both small- and large-scale components but suppresses at medium scales. This enables a superdiffusive propagator to jump to remote positions without traversing intermediate locations.

Extended Data Fig. 6 Mean displacement scales linearly on a log-log plot.

The data and simulations from Fig. 6 are plotted on a logarithmic scales demonstrating the linear relationship between mean spatial displacement and time intervals. Mean displacement (MD) measures the ratio between the spatial displacement and time interval of sequence generation and can be used to infer whether a generative sampling process is operating in a diffusive or superdiffusive regime based on sampled sequences. In particular, the MD function has a slope proportional to α−1 with respect to the time interval Δt (Methods). Error bars ± s.e.m..

Extended Data Fig. 7 Sampled sequences exhibit distinctive patterns across spectral modulation regimes.

Each column shows five sample sequences generated in one of the three sampling regimes studied in Fig. 7. Note that diffusions (left column) tend to remain within a single clique. Superdiffusions (middle column) interleave local diffusions with occasional jumps to other cliques. Jumps are observed as lines crossing the center of the ring which do not conform to any transition step in the underlying state-space structure (gray lines). Minimally autocorrelating propagators (right column) repeatedly generate large jumps between cliques.

Supplementary information

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

McNamee, D.C., Stachenfeld, K.L., Botvinick, M.M. et al. Flexible modulation of sequence generation in the entorhinal–hippocampal system. Nat Neurosci 24, 851–862 (2021). https://doi.org/10.1038/s41593-021-00831-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue date:

  • DOI: https://doi.org/10.1038/s41593-021-00831-7

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing