Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Visual odometry with neuromorphic resonator networks

Nature Machine Intelligence volume 6, pages 653–663 (2024)Cite this article

Subjects

A preprint version of the article is available at arXiv.

Abstract

Visual odometry (VO) is a method used to estimate self-motion of a mobile robot using visual sensors. Unlike odometry based on integrating differential measurements that can accumulate errors, such as inertial sensors or wheel encoders, VO is not compromised by drift. However, image-based VO is computationally demanding, limiting its application in use cases with low-latency, low-memory and low-energy requirements. Neuromorphic hardware offers low-power solutions to many vision and artificial intelligence problems, but designing such solutions is complicated and often has to be assembled from scratch. Here we propose the use of vector symbolic architecture (VSA) as an abstraction layer to design algorithms compatible with neuromorphic hardware. Building from a VSA model for scene analysis, described in our companion paper, we present a modular neuromorphic algorithm that achieves state-of-the-art performance on two-dimensional VO tasks. Specifically, the proposed algorithm stores and updates a working memory of the presented visual environment. Based on this working memory, a resonator network estimates the changing location and orientation of the camera. We experimentally validate the neuromorphic VSA-based approach to VO with two benchmarks: one based on an event-camera dataset and the other in a dynamic scene with a robotic task.

Access through your institution
Buy or subscribe

This is a preview of subscription content, access via your institution

Access options

Access through your institution

Buy this article

  • Purchase on SpringerLink
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Neuromorphic event-based visual odometry with the hierarchical resonator network.
Fig. 2: Tracking of the camera rotation from the event-based shapes_rotation dataset in simulation.
Fig. 3: Tracking of the camera rotation from the event-based shapes_rotation dataset in simulation using both IMU and event-based vision sensors.
Fig. 4: Tracking of the location and rotation of an event-based camera mounted on a robotic arm.

Similar content being viewed by others

Data availability

The event-based shapes dataset41 is publicly available at https://rpg.ifi.uzh.ch/davis_data.html. The robotic arm data generated and analysed during the current study are available via Code Ocean at https://doi.org/10.24433/CO.6568112.v1 (ref. 73).

Code availability

The source code to demonstrate the hierarchical resonator on the VO task73 is available via Code Ocean at https://doi.org/10.24433/CO.6568112.v1.

References

  1. Srinivasan, M. Honey bees as a model for vision, perception, and cognition. Annu. Rev. Entomol. 55, 267–284 (2010).

    Article  Google Scholar 

  2. Nistér, D., Naroditsky, O. & Bergen, J. Visual odometry. In Proc. 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Vol. 1, 652–659 (IEEE, 2004).

  3. Scaramuzza, D. & Fraundorfer, F. Visual odometry [tutorial]. IEEE Robot. Autom. Mag. 18, 80–92 (2011).

    Article  Google Scholar 

  4. Palossi, D. et al. A 64-mW DNN-based visual navigation engine for autonomous nano-drones. IEEE Internet Things J. 6, 8357–8371 (2019).

    Article  Google Scholar 

  5. Moravec, H. P. Obstacle Avoidance and Navigation in the Real World by a Seeing Robot Rover. PhD thesis, Stanford Univ. (1980).

  6. Lacroix, S., Mallet, A., Chatila, R. & Gallo, L. Rover self localization in planetary-like environments. In Artificial Intelligence, Robotics and Automation in Space (ed Perry, M.) 433 (ESA, 1999).

  7. Corke, P., Strelow, D. & Singh, S. Omnidirectional visual odometry for a planetary rover. IEEE/RSJ International Conference on Intelligent Robots and Systems 4, 4007–4012 (IEEE, 2004).

  8. Mead, C. Neuromorphic electronic systems. Proc. IEEE 78, 1629–1636 (1990).

    Article  Google Scholar 

  9. Chicca, E., Stefanini, F., Bartolozzi, C. & Indiveri, G. Neuromorphic electronic circuits for building autonomous cognitive systems. Proc. IEEE 102, 1367–1388 (2014).

    Article  Google Scholar 

  10. Davies, M. et al. Advancing neuromorphic computing with Loihi: a survey of results and outlook. Proc. IEEE 109, 911–934 (2021).

    Article  Google Scholar 

  11. Indiveri, G. & Liu, S.-C. Memory and information processing in neuromorphic systems. Proc. IEEE 103, 1379–1397 (2015).

    Article  Google Scholar 

  12. Plate, T. A. Analogy retrieval and processing with distributed vector representations. Expert Syst. 17, 29–40 (2000).

    Article  Google Scholar 

  13. Gayler, R. Multiplicative binding, representation operators & analogy [Workshop poster]. In Advances in analogy research: integration of theory and data from the cognitive, computational, and neural sciences (New Bulgarian Univ., 1998); http://cogprints.org/502/

  14. Kanerva, P. Binary spatter-coding of ordered K-tuples. In International Conference on Artificial Neural Networks (eds Malsburg, C. et al.) 869–873 (Springer, 1996).

  15. Eliasmith, C. & Anderson, C. H. Neural Engineering: Computation, Representation, and Dynamics in Neurobiological Systems (MIT, 2003).

  16. Sandamirskaya, Y. Dynamic neural fields as a step toward cognitive neuromorphic architectures. Front. Neurosci. 7, 276 (2013).

    Google Scholar 

  17. Kleyko, D. et al. Vector symbolic architectures as a computing framework for nanoscale hardware. Proc. IEEE 110.10, 1538–1571 (2022).

    Article  Google Scholar 

  18. Liang, D. et al. Neural state machines for robust learning and control of neuromorphic agents. IEEE J. Emerg. Sel. Topics Circuits Syst. 9, 679–689 (2019).

    Article  Google Scholar 

  19. Gayler, R. W. Vector symbolic architectures answer Jackendoff’s challenges for cognitive neuroscience. In Joint International Conference on Cognitive Science (ed Slezak, P.) 133–138 (ICCS, 2003).

  20. Kanerva, P. Hyperdimensional computing: an introduction to computing in distributed representation with high-dimensional random vectors. Cogn. Comput. 1, 139–159 (2009).

    Article  Google Scholar 

  21. Plate, T. A. Holographic reduced representations. IEEE Trans. Neural. Netw. 6, 623–641 (1995).

    Article  Google Scholar 

  22. Frady, E. P. & Sommer, F. T. Robust computation with rhythmic spike patterns. Proc. Natl Acad. Sci. USA 116, 18050–18059 (2019).

    Article  Google Scholar 

  23. Renner, A. et al. Neuromorphic visual scene understanding with resonator networks. Preprint at https://arxiv.org/abs/2208.12880 (2022).

  24. Noest, A. Phasor neural networks. In Neural Information Processing Systems (ed. Anderson, D.) 584 (NIPS, 1988).

  25. Frady, P., Kanerva, P. & Sommer, F. A framework for linking computations and rhythm-based timing patterns in neural firing, such as phase precession in hippocampal place cells. In Proc. Conference on Cognitive Computational Neuroscience (CCN, 2019).

  26. Frady, E., Kleyko, D., Kymn, C., Olshausen, B. & Sommer, F. Computing on functions using randomized vector representations. Preprint at https://arxiv.org/abs/2109.03429 (2021).

  27. Komer, B., Stewart, T., Voelker, A. & Eliasmith, C. A neural representation of continuous space using fractional binding. In Annual Meeting of the Cognitive Science Society (eds Goel et al.) 2038–2043 (Cognitive Science Society, 2019).

  28. Frady, E. P., Kent, S. J., Olshausen, B. A. & Sommer, F. T. Resonator networks, 1: an efficient solution for factoring high-dimensional, distributed representations of data structures. Neural Comput. https://doi.org/10.1162/neco_a_01331 (2020).

  29. Kent, S. J., Frady, E. P., Sommer, F. T. & Olshausen, B. A. Resonator networks, 2: factorization performance and capacity compared to optimization-based methods. Neural Comput. 32, 2332–2388 (2020).

    Article  MathSciNet  Google Scholar 

  30. Casasent, D. & Psaltis, D. Position, rotation, and scale invariant optical correlation. App. Opt. 15, 1795–1799 (1976).

    Article  Google Scholar 

  31. Chen, Q. S., Defrise, M. & Deconinck, F. Symmetric phase-only matched filtering of Fourier–Mellin transforms for image registration and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 16, 1156–1168 (1994).

    Article  Google Scholar 

  32. Reddy, B. S. & Chatterji, B. N. An fft-based technique for translation, rotation, and scale-invariant image registration. IEEE Trans. Image Process. 5, 1266–1271 (1996).

    Article  Google Scholar 

  33. Gallego, G. et al. Event-based vision: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 44, 154–180 (2020).

    Article  Google Scholar 

  34. Reinbacher, C., Munda, G. & Pock, T. Real-time panoramic tracking for event cameras. In IEEE International Conference on Computational Photography 1–9 (IEEE, 2017).

  35. Rebecq, H., Horstschaefer, T. & Scaramuzza, D. Real-time visual-inertial odometry for event Cameras using keyframe-based nonlinear optimization. In Proc. British Machine Vision Conference 16 (British Machine Vision Association, 2017).

  36. Zihao Zhu, A., Atanasov, N. & Daniilidis, K. Event-based visual inertial odometry. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 5391–5399 (IEEE, 2017).

  37. Vidal, A. R., Rebecq, H., Horstschaefer, T. & Scaramuzza, D. Ultimate SLAM? Combining events, images, and IMU for robust visual SLAM in HDR and high-speed scenarios. IEEE Robot. Autom. Lett. 3, 994–1001 (2018).

  38. Nguyen, A., Do, T. T., Caldwell, D. G. & Tsagarakis, N. G. Real-time 6DOF pose relocalization for event cameras with stacked spatial LSTM networks. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops 1638–1645 (IEEE, 2019).

  39. Rebecq, H., Ranftl, R., Koltun, V. & Scaramuzza, D. High speed and high dynamic range video with an event camera. IEEE Trans. Pattern Anal. Mach. Intell. 43, 1964–1980 (2019).

    Article  Google Scholar 

  40. Xiao, K., Wang, G., Chen, Y., Xie, Y., Li, H., & Li, S. Research on event accumulator settings for event-based SLAM. In 6th International Conference on Robotics, Control and Automation (ICRCA) 50–56 (IEEE, 2022).

  41. Mueggler, E., Rebecq, H., Gallego, G., Delbruck, T. & Scaramuzza, D. The event-camera dataset and simulator: event-based data for pose estimation, visual odometry, and slam. Int. J. Robot. Res. 36, 142–149 (2017).

    Article  Google Scholar 

  42. Cook, M., Gugelmann, L., Jug, F., Krautz, C. & Steger, A. Interacting maps for fast visual interpretation. In International Joint Conference on Neural Networks 770–776 (IEEE, 2011).

  43. Kim, H., Handa, A., Benosman, R., Leng, S. H. & Davison, A. Simultaneous mosaicing and tracking with an event camera. In Proc. British Machine Vision Conference (eds Valstar, M., French, A. & Pridmore, T.) 26.1–26.12 (British Machine Vision Association, 2014).

  44. Mueggler, E., Huber, B. & Scaramuzza, D. Event-based, 6-DOF pose tracking for high-speed maneuvers. In IEEE/RSJ International Conference on Intelligent Robots and Systems (eds Papanikolopoulos, N. et al.) 2761–2768 (IEEE, 2014).

  45. Censi, A. & Scaramuzza, D. Low-latency event-based visual odometry. In IEEE International Conference on Robotics and Automation 703–710 (IEEE, 2014).

  46. Weikersdorfer, D., Adrian, D. B., Cremers, D. & Conradt, J. Event-based 3D SLAM with a depth-augmented dynamic vision sensor. In IEEE International Conference on Robotics and Automation 359–364 (IEEE, 2014).

  47. Gallego, G., Forster, C., Mueggler, E. & Scaramuzza, D. Event-based camera pose tracking using a generative event model. Preprint at https://arxiv.org/abs/1510.01972 (2015).

  48. Kueng, B., Mueggler, E., Gallego, G. & Scaramuzza, D. Low-latency visual odometry using event-based feature tracks. In IEEE/RSJ International Conference on Intelligent Robots and Systems 16–23 (IEEE, 2016).

  49. Rebecq, H., Gallego, G., Mueggler, E. & Scaramuzza, D. EMVS: Event-based multi-view stereo—3D reconstruction with an event camera in real-time. Int. J. Comput. Vis. 126, 1–21 (2017).

    Google Scholar 

  50. Rebecq, H., Horstschäfer, T., Gallego, G. & Scaramuzza, D. Evo: a geometric approach to event-based 6-DoF parallel tracking and mapping in real time. IEEE Rob. Autom. Lett. 2, 593–600 (2016).

    Article  Google Scholar 

  51. Gallego, G., Rebecq, H. & Scaramuzza, D. A unifying contrast maximization framework for event cameras, with applications to motion, depth, and optical flow estimation. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 3867–3876 (IEEE, 2018).

  52. Zhu, A. Z., Yuan, L., Chaney, K. & Daniilidis, K. Unsupervised event-based learning of optical flow, depth, and egomotion. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 989–997 (IEEE, 2019).

  53. Davies, M. et al. Loihi: a neuromorphic manycore processor with on-chip learning. IEEE Micro 38, 82–99 (2018).

    Article  Google Scholar 

  54. Esser, S. et al. Convolutional networks for fast, energy-efficient neuromorphic computing. Proc. Natl Acad. Sci. USA 113, 11441–11446 (2016).

    Article  Google Scholar 

  55. Frenkel, C., Lefebvre, M., Legat, J. D. & Bol, D. A 0.086-mm2 12.7-pJ/SOP 64k-synapse 256-neuron online-learning digital spiking neuromorphic processor in 28-nm CMOS. IEEE Trans. Biomed. Circuits Syst. 13, 145–158 (2018).

    Google Scholar 

  56. Shrestha, S. B. & Orchard, G. Slayer: spike layer error reassignment in time. In Advances in Neural Information Processing Systems (eds Wallach, H. et al.) 1412–1421 (MIT, 2018).

  57. Renner, A., Sheldon, F., Zlotnik, A., Tao, L. & Sornborger, A. The backpropagation algorithm implemented on spiking neuromorphic hardware. Preprint at https://arxiv.org/abs/2106.07030 (2021).

  58. Frady, E. P. et al. Neuromorphic nearest neighbor search using Intel’s Pohoiki springs. In Proc. Neuro-Inspired Computational Elements Workshop 1–10 (ACM, 2020).

  59. Serrano-Gotarredona, R. et al. CAVIAR: a 45k neuron, 5M synapse, 12G connects/s AER hardware sensory-processing- learning-actuating system for high-speed visual object recognition and tracking. IEEE Trans. Neural Net. 20, 1417–1438 (2009).

    Article  Google Scholar 

  60. Kendall, A., Grimes, M. & Cipolla, R. PoseNet: a convolutional network for real-time 6-DOF camera relocalization. In Proc. IEEE International Conference on Computer Vision (ICCV) 2938–2946 (IEEE, 2015).

  61. Kendall, A. & Cipolla, R. Modelling uncertainty in deep learning for camera relocalization. In 2016 International Conference on Robotics and Automation 4762–4769 (IEEE, 2016).

  62. Walch, F. et al. Image-based localization using LSTMs for structured feature correlation. In Proc. IEEE International Conference on Computer Vision (ICCV) 627–637 (IEEE, 2017).

  63. Laskar, Z., Melekhov, I., Kalia, S. & Kannala, J. Camera relocalization by computing pairwise relative poses using convolutional neural network. In Proc. IEEE International Conference on Computer Vision (ICCV) Workshops 920–929 (IEEE, 2017).

  64. Arathorn, D. W. Map-Seeking Circuits in Visual Cognition: A Computational Mechanism for Biological and Machine Vision (Stanford Univ. Press, 2002).

  65. Vogel, C. R., Arathorn, D. W., Roorda, A. & Parker, A. Retinal motion estimation in adaptive optics scanning laser ophthalmoscopy. Opt. Express 14, 487–497 (2006).

    Article  Google Scholar 

  66. Bowman, S. L., Atanasov, N., Daniilidis, K. & Pappas, G. J. Probabilistic data association for semantic SLAM. In IEEE International Conference on Robotics and Automation 1722–1729 (IEEE, 2017).

  67. Brandli, C., Berner, R., Yang, M., Liu, S. C. & Delbruck, T. A 240 x 180 130 dB 3 μs latency global shutter spatiotemporal vision sensor. IEEE J. Solid-State Circuits 49, 2333–2341 (2014).

    Article  Google Scholar 

  68. Plate, T. Distributed Representations and Nested Compositional Structure. PhD thesis, Univ. of Toronto (1994).

  69. Mitrokhin, A., Sutor, P., Fermüller, C. & Aloimonos, Y. Learning sensorimotor control with neuromorphic sensors: toward hyperdimensional active perception. Sci. Robot. 4, eaaw6736 (2019).

    Article  Google Scholar 

  70. Renner, A., Evanusa, M. & Sandamirskaya, Y. Event-based attention and tracking on neuromorphic hardware. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshop 1709–1716 (IEEE, 2019).

  71. Umeyama, S. Least-squares estimation of transformation parameters between two point patterns. IEEE Trans. Pattern Anal. Mach. Intell. 13, 376–380 (1991).

    Article  Google Scholar 

  72. Zhang, Z. & Scaramuzza, D. A tutorial on quantitative trajectory evaluation for visual (-inertial) odometry. In IEEE International Conference on Intelligent Robots and Systems (eds Maciejewski, A. et al.) 7244–7251 (IEEE, 2018).

  73. Renner, A., Supic, L. & Frady, E. P. Code for visual odometry with neuromorphic resonator networks. Code Ocean https://doi.org/10.24433/CO.6568112.v1 (2024).

Download references

Acknowledgements

A.R. thanks his former students C. Nauer, A. Bojic, R. P. Belizón, M. Graetz and A. Collins for helpful discussions. Y.S. and A.R. disclose support for the research of this work from the Swiss National Science Foundation (SNSF) (ELMA PZOOP2 168183). A.R. discloses support for the research of this work from Accenture Labs, the University of Zurich postdoc grant (FK-21-136) and the VolkswagenStiftung (CLAM 9C854). F.T.S. discloses support for the research of this work from NIH (1R01EB026955-01).

Author information

Authors and Affiliations

  1. Institute of Neuroinformatics, University of Zurich and ETH Zurich, Zürich, Switzerland

    Alpha Renner & Giacomo Indiveri

  2. Forschungszentrum Jülich, Jülich, Germany

    Alpha Renner

  3. Accenture Labs, San Francisco, CA, USA

    Lazar Supic & Andreea Danielescu

  4. Neuromorphic Computing Lab, Intel Labs, Santa Clara, CA, USA

    E. Paxon Frady & Friedrich T. Sommer

  5. Redwood Center for Theoretical Neuroscience, UC Berkeley, Berkeley, CA, USA

    Friedrich T. Sommer

  6. Intel Labs, Zürich, Switzerland

    Yulia Sandamirskaya

  7. ZHAW Zurich University of Applied Sciences, Wädenswil, Switzerland

    Yulia Sandamirskaya

Authors
  1. Alpha Renner
    View author publications

    Search author on:PubMed Google Scholar

  2. Lazar Supic
    View author publications

    Search author on:PubMed Google Scholar

  3. Andreea Danielescu
    View author publications

    Search author on:PubMed Google Scholar

  4. Giacomo Indiveri
    View author publications

    Search author on:PubMed Google Scholar

  5. E. Paxon Frady
    View author publications

    Search author on:PubMed Google Scholar

  6. Friedrich T. Sommer
    View author publications

    Search author on:PubMed Google Scholar

  7. Yulia Sandamirskaya
    View author publications

    Search author on:PubMed Google Scholar

Contributions

A.R., L.S., A.D., G.I., E.P.F., F.T.S. and Y.S. contributed to writing and editing of the paper. Y.S. and L.S. conceptualized the project in the robotic space. A.R., E.P.F. and F.T.S. conceptualized the project in the algorithmic space. A.R. developed the VO network model and performed and analysed the network simulations. L.S. performed the robotic arm experiments.

Corresponding authors

Correspondence to Alpha Renner, Lazar Supic, Friedrich T. Sommer or Yulia Sandamirskaya.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Machine Intelligence thanks Yiannis Aloimonos and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1 Hierarchical resonator for visual odometry.

Colors match Fig. 1. The dynamics are explained in Eq. (4).

Supplementary information

Reporting Summary

Supplementary Video 1

The video visualizes the tracking of the location and rotation of an event-based camera mounted on a robotic arm and supplements Fig. 4d. The input to the network (pre-processed events from an event-based camera) is shown in green, and the transformed readout from the map vector in red. The yellow pixels indicate overlap between the map and camera view. The map is transformed into the input reference frame using the HRN’s camera pose estimate (output). The bowl is removed around second 25 and subsequently fades from the map. The video is illustrative and does not contain frames for all iterations of the network.

Supplementary Video 2

The video visualizes the dynamics of the map and supplements Fig. 4d. Each frame shows the readout from the map vector, that is, it visualizes the similarity of each pixel’s codebook vector with the map vector. Higher similarity is shown in brighter colours. The bowl is removed around second 25 and subsequently fades from the map. The video is illustrative and does not contain frames for all iterations of the network.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Renner, A., Supic, L., Danielescu, A. et al. Visual odometry with neuromorphic resonator networks. Nat Mach Intell 6, 653–663 (2024). https://doi.org/10.1038/s42256-024-00846-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue date:

  • DOI: https://doi.org/10.1038/s42256-024-00846-2

This article is cited by

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics