Abstract
Real-time perception of dynamic visual scenes requires efficient extraction of spatiotemporal features. However, conventional image sensors fail to capture inter-pixel correlations, leading to redundant data transfer, high power consumption and latency. Here, we present a non-pixelated in-materia retinomorphic sensor (IMRS) that exploits the intrinsic spatiotemporal dynamics and correlated distributions of photocarriers for visual information processing. Built on a large-area graphene/silicon heterostructure, the IMRS integrates circumferentially arranged sampling electrodes that harness the lateral photovoltaic effect to convert incident optical patterns into spatial carrier distributions, which are further encoded as object-shape-dependent photovoltages. Mimicking the lateral inhibition of biological retinas, this sensor enables in-sensor spatiotemporal perception without image reconstruction. We demonstrate human motion recognition with over 98% accuracy while compressing raw visual data from 10,000 to 48 bytes, reducing postprocessing networks parameters by two orders of magnitude. These results establish spatiotemporal photocarrier dynamics in low-dimensional heterostructures as a computational primitive for energy-efficient, ultralow-latency processing of high-dimensional spatiotemporal information.
Similar content being viewed by others
Data availability
The source data generated in this study have been deposited in the figshare under accession code https://doi.org/10.6084/m9.figshare.31042474.
References
Kong, L., Huang, D. & Wang, Y. Long-term action dependence-based hierarchical deep association for multi-athlete tracking in sports videos. IEEE Trans. Image Process. 29, 7957–7969 (2020).
Lu, W.-L., Ting, J.-A., Little, J. J. & Murphy, K. P. Learning to track and identify players from broadcast sports videos. IEEE Trans. Pattern Anal. Mach. Intell. 35, 1704–1716 (2013).
Balter, M. L., Chen, A. I., Maguire, T. J. & Yarmush, M. L. Adaptive kinematic control of a robotic venipuncture device based on stereo vision, ultrasound, and force guidance. IEEE Trans. Ind. Electron. 64, 1626–1635 (2017).
Montijano, E., Aragues, R. & Saguees, C. Distributed data association in robotic networks with cameras and limited communications. IEEE Trans. Rob. 29, 1408–1423 (2013).
Gunel, S. et al. DeepFly3D, a deep learning-based approach for 3D limb and appendage tracking in tethered, adult Drosophila. eLife 8, e48571 (2019).
Wolff, C. et al. Multi-view light-sheet imaging and tracking with the MaMuT software reveals the cell lineage of a direct developing arthropod limb. eLife 7, e34410 (2018).
Bastug, E., Bennis, M., Medard, M. & Debbah, M. Toward interconnected virtual reality: opportunities, challenges, and enablers. IEEE Commun. Mag. 55, 110–117 (2017).
Xiong, J., Hsiang, E.-L., He, Z., Zhan, T. & Wu, S.-T. Augmented reality and virtual reality displays: emerging technologies and future perspectives. Light Sci. Appl. 10, 216 (2021).
Semnani, S. H. & Basir, O. A. Semi-flocking algorithm for motion control of mobile sensors in large-scale surveillance systems. IEEE Trans. Cybern. 45, 129–137 (2015).
Zhang, X., Wu, H., Wu, M. & Wu, C. Extended motion diffusion-based change detection for airport ground surveillance. IEEE Trans. Image Process. 29, 5677–5686 (2020).
Yuan, S. et al. Geometric deep optical sensing. Science 379, eade1220 (2023).
Han, J., Shao, L., Xu, D. & Shotton, J. Enhanced computer vision with Microsoft Kinect sensor: a review. IEEE Trans. Cybern. 43, 1318–1334 (2013).
Wiriyathammabhum, P., Summers-Stay, D., Fermuller, C. & Aloimonos, Y. Computer vision and natural language processing: recent approaches in multimedia and robotics. ACM Comput. Surv. 49, 1-44 (2017).
Yang, Y. et al. In-sensor dynamic computing for intelligent machine vision. Nat. Electron. 7, 225–233 (2024).
Pan, X. et al. Parallel perception of visual motion using light-tunable memory matrix. Sci. Adv. 9, eadi4083-eadi4083 (2023).
Zhou, Y. et al. Computational event-driven vision sensors for in-sensor spiking neural networks. Nat. Electron. 6, 870–878 (2023).
Chen, J. et al. Optoelectronic graded neurons for bioinspired in-sensor motion perception. Nat. Nanotechnol. 18, 882–888 (2023).
Zhang, Z. et al. All-in-one two-dimensional retinomorphic hardware device for motion detection and recognition. Nat. Nanotechnol. 17, 27–32 (2022).
Mahowald, M. A. & Mead, C. The silicon retina. Sci. Am. 264, 76–83 (1991).
Choi, C. et al. Reconfigurable heterogeneous integration using stackable chips with embedded artificial intelligence. Nat. Electron. 5, 386–393 (2022).
Cook, P. B. & McReynolds, J. S. Lateral inhibition in the inner retina is important for spatial tuning of ganglion cells. Nat. Neurosci. 1, 714–719 (1998).
Masland, R. H. The functional architecture of the retina. Sci. Am. 255, 102–111 (1986).
Field, G. D. et al. Spatial properties and functional organization of small bistratified ganglion cells in primate retina. J. Neurosci. 27, 13261 (2007).
Wallmark, J. T. A new semiconductor photocell using lateral photoeffect. Proc. Inst. Radio Eng. 45, 474–483 (1957).
Kaspar, C., Ravoo, B. J., van der Wiel, W. G., Wegner, S. V. & Pernice, W. H. P. The rise of intelligent matter. Nature 594, 345–355 (2021).
Liu, Y. et al. Van der Waals heterostructures and devices. Nat. Rev. Mater. 1, 16042 (2016).
Milano, G. et al. In materia reservoir computing with a fully memristive architecture based on self-organizing nanowire networks. Nat. Mater. 21, 195–202 (2022).
Chen, T. et al. Classification with a disordered dopant-atom network in silicon. Nature 577, 341–345 (2020).
Yue, S. et al. High ambipolar mobility in cubic boron arsenide revealed by transient reflectivity microscopy. Science 377, 433–436 (2022).
Yoshioka, K. et al. Ultrafast intrinsic optical-to-electrical conversion dynamics in a graphene photodetector. Nat. Photonics 16, 718–723 (2022).
Zeng, Z. et al. Manipulating picosecond photoresponse in van der Waals heterostructure photodetectors. Adv. Funct. Mater. 32, 2200973 (2022).
Qian, H. & Dowling, J. E. Novel GABA responses from rod- driven retinal horizontal cells. Nature 361, 162–164 (1993).
Mills, S. L. & Massey, S. C. Differential properties of two gap junctional pathways made by AII amacrine cells. Nature 377, 734–737 (1995).
Wang, H., Zhang, C., Chan, W., Tiwari, S. & Rana, F. Ultrafast response of monolayer molybdenum disulfide photodetectors. Nat. Commun. 6, 8831 (2015).
Youngblood, N. & Li, M. Ultrafast photocurrent measurements of a black phosphorus photodetector. Appl. Phys. Lett. 110, 051102 (2017).
Kim, D. et al. in 2024 IEEE International Solid-State Circuits Conference (ISSCC). 118-120.
Park, S. et al. in 2022 IEEE International Solid-State Circuits Conference (ISSCC). 1-3.
Liu, H., Wang, C., Zuo, Z., Liu, D. & Luo, J. Direct visualization of exciton transport in defective few-layer WS2 by ultrafast microscopy. Adv. Mater. 32, 1906540 (2020).
Kulig, M. et al. Exciton diffusion and halo effects in monolayer semiconductors. Phys. Rev. Lett. 120, 207401 (2018).
Drake, G. W. Springer handbook of atomic, molecular, and optical physics. (Springer Science & Business Media, 2007).
Neamen, D. A. & Biswas, D. Semiconductor physics and devices. 206-219 (McGraw-Hill Higher Education, 2011).
Wang, W., Lu, J. & Ni, Z. Position-sensitive detectors based on two-dimensional materials. Nano Res. 14, 1889–1900 (2021).
Liu, K. et al. A time-division position-sensitive detector image system for high-speed multitarget trajectory tracking. Adv. Mater. 34, 2206638 (2022).
Liu, K. et al. Graphene-based infrared position-sensitive detector for precise measurements and high-speed trajectory tracking. Nano Lett. 19, 8132–8137 (2019).
Acknowledgements
This work was supported in part by the National Key Research and Development Program of China (Grant No. 2024YFE0217500), the National Natural Science Foundation of China (Grant Nos. 62404099, 62174026, 62225404, 62204037, 62034004, 92464303 and T2321002), the China Postdoctoral Science Foundation (2024M760424), the Natural Science Foundation of Jiangsu Province, Major Project (Grant No BK20222007, BK20233001), the Basic Research Program of Jiangsu (Grant No BK20251300), and Leading-edge Technology Program of Jiangsu Natural Science Foundation (BK20232004).
Author information
Authors and Affiliations
Contributions
K.Y.L. and P.F.W. conceived the concept and designed the experiments. J.P.L., Z.H.N., S.J.L., and F.M. supervised the entire project. K.Y.L. carried out device fabrication. K.Y.L., D.L.G., P.Y.Z., and X.L.Z. were involved in the electrical and photoresponse measurements. K.Y.L., D.Y.W., T.Z. (Tao Zhou) and T.Z. (Ting Zheng) helped with the pump-probe transient microscopy setup and analysis. S.J.L., P.F.W., and W.H.W. carried out the human motion posture collection and analyzed the data. P.F.W., Y.Z., and C.P. designed the recurrent neural network for human motion recognition. K.Y.L., P.F.W., J.P.L., Z.H.N., S.J.L., and F.M. wrote the paper, with all the authors contributing to the discussion and preparation of the manuscript.
Corresponding authors
Ethics declarations
Competing interests
The authors declare no competing interests.
Peer review
Peer review information
Nature Communications thanks the anonymous reviewer(s) for their contribution to the peer review of this work. A peer review file is available.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
About this article
Cite this article
Liu, K., Wang, P., Zhou, T. et al. Non-pixelated in-materia retinomorphic sensor via photocarrier dynamics for precise spatiotemporal perception. Nat Commun (2026). https://doi.org/10.1038/s41467-026-72104-5
Received:
Accepted:
Published:
DOI: https://doi.org/10.1038/s41467-026-72104-5
