Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Reconfigurable in-sensor processing based on a multi-phototransistor–one-memristor array

Nature Electronics volume 7, pages 991–1003 (2024)Cite this article

Subjects

Abstract

Memristors with photonic sensory capabilities can be used as elements in machine vision systems but face challenges in terms of encoding and processing optical data. This has led to different neural network architectures being developed for specific vision tasks, which limits progress towards more versatile in-sensor vision computing platforms. Here we describe a multi-phototransistor and one-memristor array that is based on niobium oxide memristors. It has reconfigurable dynamics and is compatible with both machine learning (analogue) and bioinspired (spiking) neural network architectures. The array can sense and process optical images and synchronize spatio-temporal data across different encoding formats. When the array is coupled with a classifier network using a one-transistor and one-memristor non-volatile memory array, it supports a variety of optical neural networks (including optical convolutional neural networks, recurrent neural networks and spiking neural networks). The resulting system can perform various computing vision tasks, such as recognizing static, motion and colour images.

Access through your institution
Buy or subscribe

This is a preview of subscription content, access via your institution

Access options

Access through your institution

Buy this article

  • Purchase on SpringerLink
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Neuromorphic vision computing with versatile neural network architectures using a reconfigurable MP1R array.
Fig. 2: MP1R array with configurable vision sensory functions.
Fig. 3: Hardware configurable in-sensor vision computing system based on MP1R and non-volatile hafnia memristor array.
Fig. 4: OCNN based on the MP1R- and 1T1R NVM array for image recognition.
Fig. 5: Optical in-sensor reservoir neural network using short-term memory for event image recognition.
Fig. 6: OSNN with optical sensory spiking neurons for image recognition.

Similar content being viewed by others

Data availability

Source data are available via Zenodo at https://doi.org/10.5281/zenodo.13747890 (ref. 56). Source data are provided with this paper. Other data that support the findings of this study are available from the corresponding author upon reasonable request.

Code availability

The codes used for the simulations are available via Zenodo at https://doi.org/10.5281/zenodo.13747890 (ref. 56). Other codes used for data plotting are available from the corresponding author upon reasonable request.

References

  1. Gu, L. et al. A biomimetic eye with a hemispherical perovskite nanowire array retina. Nature 581, 278–282 (2020).

    Article  Google Scholar 

  2. Wu, J. et al. An integrated imaging sensor for aberration-corrected 3D photography. Nature 612, 62–71 (2022).

    Article  Google Scholar 

  3. Long, Z. et al. A neuromorphic bionic eye with filter-free color vision using hemispherical perovskite nanowire array retina. Nat. Commun. 14, 1972 (2023).

    Article  Google Scholar 

  4. Merolla, P. A. et al. A million spiking-neuron integrated circuit with a scalable communication network and interface. Science 345, 668–673 (2014).

    Article  Google Scholar 

  5. Ielmini, D. & Wong, H. S. P. In-memory computing with resistive switching devices. Nat. Electron. 1, 333–343 (2018).

    Article  Google Scholar 

  6. Zidan, M. A., Strachan, J. P. & Lu, W. D. The future of electronics based on memristive systems. Nat. Electron. 1, 22–29 (2018).

    Article  Google Scholar 

  7. Sebastian, A. et al. Memory devices and applications for in-memory computing. Nat. Nanotechnol. 15, 529–544 (2020).

    Article  Google Scholar 

  8. Zhang, W. et al. Neuro-inspired computing chips. Nat. Electron. 3, 371–382 (2020).

    Article  Google Scholar 

  9. Wang, Z. et al. Resistive switching materials for information processing. Nat. Rev. Mater. 5, 173–195 (2020).

    Article  Google Scholar 

  10. Zhou, F. & Chai, Y. Near-sensor and in-sensor computing. Nat. Electron. 3, 664–671 (2020).

    Article  Google Scholar 

  11. Chai, Y. In-sensor computing. Nature 579, 32–33 (2020).

    Article  Google Scholar 

  12. Kaspar, C. et al. The rise of intelligent matter. Nature 594, 345–355 (2021).

    Article  Google Scholar 

  13. Zhou, F. et al. Optoelectronic resistive random access memory for neuromorphic vision sensors. Nat. Nanotechnol. 14, 776–782 (2019).

    Article  Google Scholar 

  14. Hu, L. et al. All-optically controlled memristor for optoelectronic neuromorphic computing. Adv. Funct. Mater. 31, 2005582 (2021).

    Article  Google Scholar 

  15. Mennel, L. et al. Ultrafast machine vision with 2D material neural network image sensors. Nature 579, 62–66 (2020).

    Article  Google Scholar 

  16. Yao, P. et al. Fully hardware-implemented memristor convolutional neural network. Nature 577, 641–646 (2020).

    Article  Google Scholar 

  17. Lee, S. et al. Programmable black phosphorus image sensor for broadband optoelectronic edge computing. Nat. Commun. 13, 1485 (2022).

    Article  Google Scholar 

  18. Zhang, Z. et al. All-in-one two-dimensional retinomorphic hardware device for motion detection and recognition. Nat. Nanotechnol. 17, 27–32 (2022).

    Article  Google Scholar 

  19. Sun, L. et al. In-sensor reservoir computing for language learning via two-dimensional memristors. Sci. Adv. 7, eabg1455 (2021).

    Article  Google Scholar 

  20. Lee, D. et al. In-sensor image memorization and encoding via optical neurons for bio-stimulus domain reduction toward visual cognitive processing. Nat. Commun. 13, 5223 (2022).

    Article  Google Scholar 

  21. Liu, K. et al. An optoelectronic synapse based on α-In2Se3 with controllable temporal dynamics for multimode and multiscale reservoir computing. Nat. Electron. 5, 761–773 (2022).

    Article  Google Scholar 

  22. Zhang, Z. et al. In-sensor reservoir computing system for latent fingerprint recognition with deep ultraviolet photo-synapses and memristor array. Nat. Commun. 13, 6590 (2022).

    Article  Google Scholar 

  23. Wu, X. et al. Wearable in-sensor reservoir computing using optoelectronic polymers with through-space charge-transport characteristics for multi-task learning. Nat. Commun. 14, 468 (2023).

    Article  Google Scholar 

  24. Wu, Q. et al. Spike encoding with optic sensory neurons enable a pulse coupled neural network for ultraviolet image segmentation. Nano Lett. 20, 8015–8023 (2020).

    Article  Google Scholar 

  25. Chen, J. et al. Optoelectronic graded neurons for bioinspired in-sensor motion perception. Nat. Nanotechnol. 18, 882–888 (2023).

    Article  Google Scholar 

  26. Wang, F. et al. A two-dimensional mid-infrared optoelectronic retina enabling simultaneous perception and encoding. Nat. Commun. 14, 1938 (2023).

    Article  Google Scholar 

  27. Wang, X. et al. Vertically integrated spiking cone photoreceptor arrays for color perception. Nat. Commun. 14, 3444 (2023).

    Article  Google Scholar 

  28. Zhou, Y. et al. Computational event-driven vision sensors for in-sensor spiking neural networks. Nat. Electron. 6, 870–878 (2023).

    Article  Google Scholar 

  29. Dang, B. et al. One-phototransistor-one-memristor array with high-linearity light-tunable weight for optic neuromorphic computing. Adv. Mater. 35, 2204844 (2022).

    Article  Google Scholar 

  30. Ma, S. et al. A 619-pixel machine vision enhancement chip based on two-dimensional semiconductors. Sci. Adv. 8, eabn9328 (2022).

    Article  Google Scholar 

  31. Pei, J. et al. Towards artificial general intelligence with hybrid Tianjic chip architecture. Nature 572, 106–111 (2019).

    Article  Google Scholar 

  32. Wang, C.-Y. et al. Gate-tunable van der Waals heterostructure for reconfigurable neural network vision sensor. Sci. Adv. 6, eaba6173 (2020).

    Article  Google Scholar 

  33. Li, T. et al. Reconfigurable, non-volatile neuromorphic photovoltaics. Nat. Nanotechnol. 18, 1303–1310 (2023).

    Article  Google Scholar 

  34. Lübben, M., Wiefels, S., Waser, R. & Valov, I. Processes and effects of oxygen and moisture in resistively switching TaOx and HfOx. Adv. Electron. Mater. 4, 1700458 (2018).

    Article  Google Scholar 

  35. Valov, I. & Tsuruoka, T. Effects of moisture and redox reactions in VCM and ECM resistive switching memories. J. Phys. D: Appl. Phys. 51, 413001 (2018).

    Article  Google Scholar 

  36. Wang, W. et al. A self-rectification and quasi-linear analogue memristor for artificial neural networks. IEEE Electron Device Lett. 40, 1407–1410 (2019).

    Article  Google Scholar 

  37. Wang, R. et al. Deep reservoir computing based on self-rectifying memristor synapse for time series prediction. Appl. Phys. Lett. 123, 042109 (2023).

    Article  Google Scholar 

  38. Konstantin, V. E. et al. Hydrogen radical enhanced atomic layer deposition of TaOx: saturation studies and methods for oxygen deficiency control. J. Mater. Chem. C 6, 9667–9674 (2018).

    Article  Google Scholar 

  39. Xu, J. H., Jarlborg, T. & Freeman, A. J. Self-consistent band structure of the rutile dioxides NbO2, RuO2, and IrO2. Phys. Rev. B 40, 7939 (1989).

    Article  Google Scholar 

  40. Aziz, J. et al. Chemical nature of electrode and the switching response of RF-sputtered NbOx films. Nanomaterials 10, 2164 (2020).

    Article  Google Scholar 

  41. Chen, W.-T., Hsueh, H.-W., Zan, H.-W. & Tsai, C.-C. Light-enhanced bias stress effect on amorphous In-Ga-Zn-O thin-film transistor with lights of varying colors. Electrochem. Solid-State Lett. 14, H297 (2011).

    Article  Google Scholar 

  42. Nahm, H. H. & Kim, Y. S. Undercoordinated indium as an intrinsic electron-trap center in amorphous InGaZnO4. NPG Asia Mater. 6, e143 (2014).

    Article  Google Scholar 

  43. Jeong, J. K. et al. Origin of threshold voltage instability in indium-gallium-zinc oxide thin film transistors. Appl. Phys. Lett. 93, 123508 (2008).

    Article  Google Scholar 

  44. Jang, H. et al. In-sensor optoelectronic computing using electrostatically doped silicon. Nat. Electron. 5, 519–525 (2022).

    Article  Google Scholar 

  45. Chen, Z. et al. All-ferroelectric implementation of reservoir computing. Nat. Commun. 14, 3585 (2023).

    Article  Google Scholar 

  46. Milano, G. et al. In materia reservoir computing with a fully memristive architecture based on self-organizing nanowire networks. Nat. Mater. 21, 195–202 (2022).

    Article  Google Scholar 

  47. Chen, Y.-L., Lu, C.-C., Juang, K.-C. & Tang, K.-T. Conversion of artificial neural network to spiking neural network for hardware implementation. In Proc. 2019 IEEE International Conference on Consumer Electronics 1–2 (IEEE, 2019).

  48. Wu, Z. et al. Threshold switching memristor-based radial-based spiking neuron circuit for conversion based spiking neural networks adversarial attack improvement. IEEE Trans. Circuits Syst. II: Express Briefs 71, 1446–1450 (2024).

    Google Scholar 

  49. Pang, B., Nijkamp, E. & Wu, Y. Deep learning with TensorFlow: a review. J. Educ. Behav. Stat. 45, 227–248 (2020).

    Article  Google Scholar 

  50. Reyad, M., Sarhan, A. & Arafa, M. A modified Adam algorithm for deep neural network optimization. Neural Comput. Appl. 35, 17095–17112 (2023).

    Article  Google Scholar 

  51. Ho, Y. & Wookey, S. The real-world-weight cross-entropy loss function: modeling the costs of mislabeling. IEEE Access 8, 4806–4813 (2020).

    Article  Google Scholar 

  52. Liu, Q. & Furber, S. (2016). Noisy softplus: a biology inspired activation function. In Neural Information Processing. ICONIP 2016. Lecture Notes in Computer Science Vol. 9950 (eds Hirose, A et al.) 405–412 (Springer, 2016).

  53. Dominey, P. F. Narrative event segmentation in the cortical reservoir. PLoS Comput. Biol. 17, e1008993 (2021).

    Article  Google Scholar 

  54. Du, C. et al. Reservoir computing using dynamic memristors for temporal information processing. Nat. Commun. 8, 2204 (2017).

    Article  Google Scholar 

  55. Ding, J. et al. Optimal ANN-SNN conversion for fast and accurate inference in deep spiking neural networks. In Proc. 30th International Joint Conference on Artificial Intelligence (ed. Zhou, Z.-H.) 2328–2336 (International Joint Conference on Artificial Intelligence Organization, 2021).

  56. Dang, B. Reconfigurable in-sensor processing based on a multi-phototransistor–one-memristor array. Zenodo https://doi.org/10.5281/zenodo.13747890 (2024).

Download references

Acknowledgements

This work was supported by the National Key R&D Program of China (Grant No. 2023YFB4502200), the National Natural Science Foundation of China (Grant Nos 61925401, 92064004, 61927901, 8206100486 and 92164302), the Beijing Natural Science Foundation (Grant No. L234026) and the 111 Project (Grant No. B18001).

Author information

Authors and Affiliations

  1. Beijing Advanced Innovation Center for Integrated Circuits, School of Integrated Circuits, Peking University, Beijing, China

    Bingjie Dang, Teng Zhang, Xulei Wu, Keqin Liu, Ru Huang & Yuchao Yang

  2. Guangdong Provincial Key Laboratory of In-Memory Computing Chips, School of Electronic and Computer Engineering, Peking University, Shenzhen, China

    Yuchao Yang

  3. Institute for Artificial Intelligence, Frontiers Science Center for Nano-optoelectronics, Peking University, Beijing, China

    Yuchao Yang

  4. Center for Brain Inspired Intelligence, Chinese Institute for Brain Research (CIBR), Beijing, Beijing, China

    Yuchao Yang

Authors
  1. Bingjie Dang
    View author publications

    Search author on:PubMed Google Scholar

  2. Teng Zhang
    View author publications

    Search author on:PubMed Google Scholar

  3. Xulei Wu
    View author publications

    Search author on:PubMed Google Scholar

  4. Keqin Liu
    View author publications

    Search author on:PubMed Google Scholar

  5. Ru Huang
    View author publications

    Search author on:PubMed Google Scholar

  6. Yuchao Yang
    View author publications

    Search author on:PubMed Google Scholar

Contributions

B.D. and Y.Y. designed the study. B.D. fabricated the MP1R array and the vision computing test system and performed the electrical measurements. T.Z. and B.D. prepared the 1T1R memristor array. B.D., T.Z., R.H. and Y.Y. developed the 1T1R non-volatile memristor array integration processes. B.D. performed the ex situ TEM characterization. B.D. demonstrated the application of OCNNs, ORNNs and OSNNs in optical image recognition. B.D., X.W., K.L. and Y.Y. prepared the manuscript. Y.Y. directed all the experimental research and supervised this work. All authors analysed the results and implications and commented on the manuscript at all stages.

Corresponding author

Correspondence to Yuchao Yang.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Electronics thanks Hongwei Tan and Ilia Valov for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Figs. 1–29, Table 1 and discussion.

Source data

Source Data Figs. 2–6

Data underlying Figs. 2c–j, 3b,c, 4g–i,k, 5d,g and 6b,d–g.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dang, B., Zhang, T., Wu, X. et al. Reconfigurable in-sensor processing based on a multi-phototransistor–one-memristor array. Nat Electron 7, 991–1003 (2024). https://doi.org/10.1038/s41928-024-01280-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue date:

  • DOI: https://doi.org/10.1038/s41928-024-01280-3

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing