Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Single-chip photonic deep neural network with forward-only training

Nature Photonics volume 18, pages 1335–1343 (2024)Cite this article

Subjects

Abstract

As deep neural networks revolutionize machine learning, energy consumption and throughput are emerging as fundamental limitations of complementary metal–oxide–semiconductor (CMOS) electronics. This has motivated a search for new hardware architectures optimized for artificial intelligence, such as electronic systolic arrays, memristor crossbar arrays and optical accelerators. Optical systems can perform linear matrix operations at an exceptionally high rate and efficiency, motivating recent demonstrations of low-latency matrix accelerators and optoelectronic image classifiers. However, demonstrating coherent, ultralow-latency optical processing of deep neural networks has remained an outstanding challenge. Here we realize such a system in a scalable photonic integrated circuit that monolithically integrates multiple coherent optical processor units for matrix algebra and nonlinear activation functions into a single chip. We experimentally demonstrate this fully integrated coherent optical neural network architecture for a deep neural network with six neurons and three layers that optically computes both linear and nonlinear functions with a latency of 410 ps, unlocking new applications that require ultrafast, direct processing of optical signals. We implement backpropagation-free in situ training on this system, achieving 92.5% accuracy on a six-class vowel classification task, which is comparable to the accuracy obtained on a digital computer. This work lends experimental evidence to theoretical proposals for in situ training, enabling orders of magnitude improvements in the throughput of training data. Moreover, the fully integrated coherent optical neural network opens the path to inference at nanosecond latency and femtojoule per operation energy efficiency.

Access through your institution
Buy or subscribe

This is a preview of subscription content, access via your institution

Access options

Access through your institution

Buy this article

  • Purchase on SpringerLink
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Architecture of FICONN.
Fig. 2: PIC.
Fig. 3: NOFU.
Fig. 4: Backpropagation-free in situ training.

Similar content being viewed by others

Data availability

All the data that support the findings of this study are included in the Article and its Supplementary Information. Source data are available via figshare at https://doi.org/10.6084/m9.figshare.27174468.

References

  1. Krizhevsky, A., Sutskever, I. & Hinton, G. E. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 1097–1105 (Curran Associates, 2012).

  2. He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 770–778 (IEEE, 2016).

  3. Brown, T. et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems 1877–1901 (Curran Associates, 2020).

  4. Mirhoseini, A. et al. A graph placement methodology for fast chip design. Nature 594, 207–212 (2021).

    Article  ADS  Google Scholar 

  5. Vinyals, O. et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature 575, 350–354 (2019).

    Article  ADS  Google Scholar 

  6. Silver, D. et al. Mastering the game of Go without human knowledge. Nature 550, 354–359 (2017).

    Article  ADS  Google Scholar 

  7. Wetzstein, G. et al. Inference in artificial intelligence with deep optics and photonics. Nature 588, 39–47 (2020).

    Article  ADS  Google Scholar 

  8. Shen, Y. et al. Deep learning with coherent nanophotonic circuits. Nat. Photon. 11, 441–446 (2017).

    Article  ADS  Google Scholar 

  9. Sludds, A. et al. Delocalized photonic deep learning on the internet’s edge. Science 378, 270–276 (2022).

    Article  ADS  Google Scholar 

  10. Feldmann, J. et al. Parallel convolutional processing using an integrated photonic tensor core. Nature 589, 52–58 (2021).

    Article  ADS  Google Scholar 

  11. Xu, X. et al. 11 TOPS photonic convolutional accelerator for optical neural networks. Nature 589, 44–51 (2021).

    Article  ADS  Google Scholar 

  12. Wang, T. et al. An optical neural network using less than 1 photon per multiplication. Nat. Commun. 13, 123 (2022).

    Article  ADS  Google Scholar 

  13. Tait, A. N., Nahmias, M. A., Shastri, B. J. & Prucnal, P. R. Broadcast and weight: an integrated network for scalable photonic spike processing. J. Light. Technol. 32, 4029–4041 (2014).

    Article  Google Scholar 

  14. Ashtiani, F., Geers, A. J. & Aflatouni, F. An on-chip photonic deep neural network for image classification. Nature 606, 501–506 (2022).

    Article  ADS  Google Scholar 

  15. Bernstein, L. et al. Single-shot optical neural network. Sci. Adv. 9, eadg7904 (2023).

    Article  Google Scholar 

  16. Liu, Z. et al. Efficient and robust LiDAR-based end-to-end navigation. In 2021 IEEE International Conference on Robotics and Automation (ICRA) 13247–13254 (IEEE Press, 2021).

  17. Messick, C. et al. Analysis framework for the prompt discovery of compact binary mergers in gravitational-wave data. Phys. Rev. D 95, 042001 (2017).

    Article  ADS  Google Scholar 

  18. Huerta, E. A. et al. Enabling real-time multi-messenger astrophysics discoveries with deep learning. Nat. Rev. Phys. 1, 600–608 (2019).

    Article  Google Scholar 

  19. Duarte, J. et al. Fast inference of deep neural networks in FPGAs for particle physics. J. Instrum. 13, P07027 (2018).

    Article  Google Scholar 

  20. Coelho, C. N. et al. Automatic heterogeneous quantization of deep neural networks for low-latency inference on the edge for particle detectors. Nat. Mach. Intell. 3, 675–686 (2021).

    Article  Google Scholar 

  21. Zibar, D., Piels, M., Jones, R. & Schaeffer, C. G. Machine learning techniques in optical communication. J. Light. Technol. 34, 1442–1452 (2016).

    Article  ADS  Google Scholar 

  22. Nahmias, M. A. et al. Photonic multiply-accumulate operations for neural networks. IEEE J. Sel. Topics Quantum Electron. 26, 7701518 (2020).

    Article  Google Scholar 

  23. Neefs, H., Heuven, P. V. & Campenhout, J. M. V. Latency requirements of optical interconnects at different memory hierarchy levels of a computer system. In Proc. SPIE 3490, Optics in Computing ’98 552–555 (SPIE, 1998).

  24. Williamson, I. A. D. et al. Reprogrammable electro-optic nonlinear activation functions for optical neural networks. IEEE J. Sel. Topics Quantum Electron. 26, 7700412 (2020).

    Article  Google Scholar 

  25. Pour Fard, M. M. et al. Experimental realization of arbitrary activation functions for optical neural networks. Opt. Express 28, 12138–12148 (2020).

    Article  ADS  Google Scholar 

  26. Tait, A. N. et al. Silicon photonic modulator neuron. Phys. Rev. Appl. 11, 064043 (2019).

    Article  ADS  Google Scholar 

  27. Nahmias, M. A. et al. An integrated analog O/E/O link for multi-channel laser neurons. Appl. Phys. Lett. 108, 151106 (2016).

    Article  ADS  Google Scholar 

  28. Miller, D. A. B. Self-configuring universal linear optical component. Photon. Res. 1, 1–15 (2013).

    Article  ADS  Google Scholar 

  29. Zhang, H. et al. An optical neural chip for implementing complex-valued neural network. Nat. Commun. 12, 457 (2021).

    Article  ADS  Google Scholar 

  30. Huang, C. et al. A silicon photonic-electronic neural network for fibre nonlinearity compensation. Nat. Electron. 4, 837–844 (2021).

    Article  Google Scholar 

  31. Konečný, J., McMahan, B. & Ramage, D. Federated optimization: distributed optimization beyond the datacenter. In 8th NIPS Workshop on Optimization for Machine Learning http://opt-ml.org/papers/OPT2015_paper_10.pdf (2015).

  32. Pai, S. et al. Experimentally realized in situ backpropagation for deep learning in photonic neural networks. Science 380, 398–404 (2023).

    Article  ADS  Google Scholar 

  33. Zhang, H. et al. Efficient on-chip training of optical neural networks using genetic algorithm. ACS Photonics 8, 1662–1672 (2021).

    Article  Google Scholar 

  34. Bueno, J. et al. Reinforcement learning in a large-scale photonic recurrent neural network. Optica 5, 756–760 (2018).

    Article  ADS  Google Scholar 

  35. Bogaerts, W. et al. Programmable photonic circuits. Nature 586, 207–216 (2020).

    Article  ADS  Google Scholar 

  36. Clements, W. R., Humphreys, P. C., Metcalf, B. J., Kolthammer, W. S. & Walsmley, I. A. Optimal design for universal multiport interferometers. Optica 3, 1460–1465 (2016).

    Article  Google Scholar 

  37. Jing, L. et al. Tunable efficient unitary neural networks (EUNN) and their application to RNNs. In Proc. Machine Learning Research 1733–1741 (PMLR, 2017).

  38. Bandyopadhyay, S., Hamerly, R. & Englund, D. Hardware error correction for programmable photonics. Optica 8, 1247–1255 (2021).

    Article  ADS  Google Scholar 

  39. Hamerly, R., Bandyopadhyay, S. & Englund, D. Stability of self-configuring large multiport interferometers. Phys. Rev. Appl. 18, 024018 (2022).

    Article  ADS  Google Scholar 

  40. Hamerly, R., Bandyopadhyay, S. & Englund, D. Accurate self-configuration of rectangular multiport interferometers. Phys. Rev. Appl. 18, 024019 (2022).

    Article  ADS  Google Scholar 

  41. Ahmed, M. G. et al. A 34Gbaud linear transimpedance amplifier with automatic gain control for 200Gb/s DP-16QAM optical coherent receivers. In 2018 Optical Fiber Communications Conference and Exposition (OFC) 1–3 (IEEE, 2018).

  42. Sedighi, B. & Scheytt, J. C. Low-power SiGe BiCMOS transimpedance amplifier for 25-GBaud optical links. IEEE Trans. Circuits Syst., II: Exp. Briefs 59, 461–465 (2012).

    Google Scholar 

  43. Miller, D. A. B. Attojoule optoelectronics for low-energy information processing and communications. J. Light. Technol. 35, 346–396 (2017).

    Article  ADS  Google Scholar 

  44. Wright, L. G. et al. Deep physical neural networks trained with backpropagation. Nature 601, 549–555 (2022).

    Article  ADS  Google Scholar 

  45. Hughes, T. W., Minkov, M., Shi, Y. & Fan, S. Training of photonic neural networks through in situ backpropagation and gradient measurement. Optica 5, 864–871 (2018).

    Article  Google Scholar 

  46. Zhou, H. et al. Self-configuring and reconfigurable silicon photonic signal processor. ACS Photonics 7, 792–799 (2020).

    Article  Google Scholar 

  47. Cauwenberghs, G. A fast stochastic error-descent algorithm for supervised learning and optimization. In Advances in Neural Information Processing Systems 244–251 (Morgan-Kaufmann, 1992).

  48. Spall, J. C. An Overview of the Simultaneous Perturbation Method for Efficient Optimization. Report No. 19 (Johns Hopkins Applied Physics Laboratory, 1998).

  49. Hillenbrand, J. M. Internet Archive (1995); https://web.archive.org/web/20221024030937/https://homepages.wmich.edu/~hillenbr/voweldata.html

  50. Micikevicius, P. et al. Mixed precision training. In International Conference on Learning Representations https://openreview.net/pdf?id=r1gs9JgRZ (ICLR, 2018).

  51. Sze, V., Chen, Y.-H., Yang, T.-J. & Emer, J. S. How to evaluate deep neural network processors: TOPS/W (alone) considered harmful. IEEE Solid-State Circuits Magazine 12, 28–41 (2020).

    Article  Google Scholar 

  52. Timurdogan, E. et al. An ultralow power athermal silicon modulator. Nat. Commun. 5, 4008 (2014).

    Article  ADS  Google Scholar 

  53. Gyger, S. et al. Reconfigurable photonics with on-chip single-photon detectors. Nat. Commun. 12, 1408 (2021).

    Article  ADS  Google Scholar 

  54. Baghdadi, R. et al. Dual slot-mode NOEM phase shifter. Opt. Express 29, 19113–19119 (2021).

    Article  ADS  Google Scholar 

  55. Edinger, P. et al. Silicon photonic microelectromechanical phase shifters for scalable programmable photonics. Opt. Lett. 46, 5671–5674 (2021).

    Article  Google Scholar 

  56. Kwon, K. et al. 128 × 128 silicon photonic MEMS switch with scalable row/column addressing. In 2018 Conference on Lasers and Electro-Optics (CLEO) SF1A.4 (Optica Publishing Group, 2018).

  57. Jouppi, N. P. et al. In-datacenter performance analysis of a tensor processing unit. In Proc. 44th Annual International Symposium on Computer Architecture 1–12 (Association for Computing Machinery, 2017).

  58. Reuer, K. et al. Realizing a deep reinforcement learning agent for real-time quantum feedback. Nat. Commun. 14, 7138 (2023).

    Article  ADS  Google Scholar 

  59. Strubell, E., Ganesh, A. & McCallum, A. Energy and policy considerations for modern deep learning research. Proc. AAAI Conf. Artif. Intell. 34, 13693–13696 (2020).

    Google Scholar 

  60. You, Y., Zhang, Z., Hsieh, C.-J., Demmel, J. & Keutzer, K. ImageNet training in minutes. In Proc. 47th International Conference on Parallel Processing 1 (Association for Computing Machinery, 2018).

  61. Jouppi, N. P. et al. A domain-specific supercomputer for training deep neural networks. Commun. ACM 63, 67–78 (2020).

    Article  Google Scholar 

  62. Li, R. et al. Silicon photonic ring-assisted MZI for 50 Gb/s DAC-less and DSP-free PAM-4 transmission. IEEE Photon. Technol. Lett. 29, 1046–1049 (2017).

    Article  ADS  Google Scholar 

  63. Shallue, C. J. et al. Measuring the effects of data parallelism on neural network training. J. Mach. Learn. Res. 20, 1–49 (2019).

    MathSciNet  Google Scholar 

  64. Akiba, T., Suzuki, S. & Fukuda, K. Extremely large minibatch SGD: training ResNet-50 on ImageNet in 15 minutes. Preprint at https://arxiv.org/abs/1711.04325 (2017).

  65. McCaughan, A. N. et al. Multiplexed gradient descent: fast online training of modern datasets on hardware neural networks without backpropagation. APL Mach. Learn. 1, 026118 (2023).

    Article  Google Scholar 

  66. Camuto, A., Willetts, M., Simsekli, U., Roberts, S. J. & Holmes, C. C. Explicit regularisation in Gaussian noise injections. In Advances in Neural Information Processing Systems 16603–16614 (Curran Associates, 2020).

  67. Liu, X., Cheng, M., Zhang, H. & Hsieh, C.-J. Towards robust neural networks via random self-ensemble. In Proc. European Conference on Computer Vision (ECCV) 369–385 (Springer, 2018).

  68. Feldmann, J., Youngblood, N., Wright, C. D., Bhaskaran, H. & Pernice, W. H. P. All-optical spiking neurosynaptic networks with self-learning capabilities. Nature 569, 208–214 (2019).

    Article  ADS  Google Scholar 

  69. López-Pastor, V. & Marquardt, F. Self-learning machines based on Hamiltonian echo backpropagation. Phys. Rev. X 13, 031020 (2023).

    Google Scholar 

  70. Sun, J., Timurdogan, E., Yaacobi, A., Hosseini, E. S. & Watts, M. R. Large-scale nanophotonic phased array. Nature 493, 195–199 (2013).

    Article  ADS  Google Scholar 

  71. Harris, N. C. et al. Accelerating artificial intelligence with silicon photonics. In Optical Fiber Communication Conference (OFC) 2020 W3A.3 (Optica Publishing Group, 2020).

  72. Shu, H. et al. Microcomb-driven silicon photonic systems. Nature 605, 457–463 (2022).

    Article  ADS  Google Scholar 

  73. Harris, N. C., Bunandar, D., Joshi, A., Basumallik, A. & Turner, R. Passage: a wafer-scale programmable photonic communication substrate. In 2022 IEEE Hot Chips 34 Symposium (HCS) 1–26 (IEEE, 2022).

  74. Blaicher, M. et al. Hybrid multi-chip assembly of optical communication engines by in situ 3D nano-lithography. Light: Sci. Appl. 9, 71 (2020).

    Article  ADS  Google Scholar 

  75. Lindenmann, N. et al. Photonic wire bonding: a novel concept for chip-scale interconnects. Opt. Express 20, 17667–17677 (2012).

    Article  ADS  Google Scholar 

  76. Flory, N. et al. Highly reliable polymer waveguide platform for multi-port photonic chip-packaging. In 2021 IEEE 71st Electronic Components and Technology Conference (ECTC) 1689–1694 (IEEE, 2021).

  77. Van Gasse, K., Wang, R. & Roelkens, G. 27 dB gain III-V-on-silicon semiconductor optical amplifier with > 17 dBm output power. Opt. Express 27, 293–302 (2019).

    Google Scholar 

  78. Davenport, M. L. et al. Heterogeneous silicon/III-V semiconductor optical amplifiers. IEEE J. Sel. Topics Quantum Electron. 22, 78–88 (2016).

    Article  ADS  Google Scholar 

  79. Liu, Y. et al. A photonic integrated circuit-based erbium-doped amplifier. Science 376, 1309–1313 (2022).

    Article  ADS  Google Scholar 

  80. Nozaki, K. et al. Femtofarad optoelectronic integration demonstrating energy-saving signal conversion and nonlinear functions. Nat. Photon. 13, 454–459 (2019).

    Article  ADS  Google Scholar 

  81. Li, G. H. et al. All-optical ultrafast ReLU function for energy-efficient nanophotonic deep learning. Nanophotonics 12, 847–855 (2022).

    Article  Google Scholar 

  82. Wang, C. et al. Integrated lithium niobate electro-optic modulators operating at CMOS-compatible voltages. Nature 562, 101–104 (2018).

    Article  ADS  Google Scholar 

  83. Pérez-López, D., López, A., DasMahapatra, P. & Capmany, J. Multipurpose self-configuration of programmable photonic circuits. Nat. Commun. 11, 6359 (2020).

    Article  ADS  Google Scholar 

  84. Mak, J. C. C., Xue, T., Yong, Z. & Poon, J. K. S. Wavelength tunable matched-pair Vernier multi-ring filters using derivative-free optimization algorithms. IEEE J. Sel. Topics Quantum Electron. 26, 5900212 (2020).

    Article  Google Scholar 

  85. Prabhu, M. et al. Accelerating recurrent Ising machines in photonic integrated circuits. Optica 7, 551–558 (2020).

    Article  Google Scholar 

  86. Pai, S., Bartlett, B., Solgaard, O. & Miller, D. A. B. Matrix optimization on universal unitary photonic devices. Phys. Rev. Appl. 11, 064044 (2019).

    Article  ADS  Google Scholar 

Download references

Acknowledgements

S.B. was supported by a National Science Foundation (NSF) Graduate Research Fellowship under grant no. 1745302, NSF award nos. 1839159 (RAISE-TAQS) and 2040695 (Convergence Accelerator), and the Air Force Office of Scientific Research (AFOSR) under award no. FA9550-20-1-0113. A.S. was supported by an NSF Graduate Research Fellowship and the aforementioned AFOSR award, as well as NSF award no. 1946976 (EAGER) and NTT Research. We would like to acknowledge P. Gaudette and D. Scott of Optelligent for packaging the PIC; R. Shi and H. Guan for feedback on the photonics layout; S. K. Vadlamani for discussions on hardware-aware training; L. Bernstein for discussions on DNN applications and feedback on the manuscript; J. Carolan and M. Prabhu for discussions on chip packaging and testing of the photonics; and D. Lewis for assistance with the use of machining tools.

Author information

Authors and Affiliations

  1. Research Laboratory of Electronics, MIT, Cambridge, MA, USA

    Saumil Bandyopadhyay, Alexander Sludds, Stefan Krastanov, Ryan Hamerly, Nicholas Harris, Darius Bunandar & Dirk Englund

  2. NTT Research Inc., Physics & Informatics Laboratories, Sunnyvale, CA, USA

    Saumil Bandyopadhyay & Ryan Hamerly

  3. Nokia Corporation, New York, NY, USA

    Matthew Streshinsky

  4. Periplous, LLC, New York, NY, USA

    Michael Hochberg

Authors
  1. Saumil Bandyopadhyay
    View author publications

    Search author on:PubMed Google Scholar

  2. Alexander Sludds
    View author publications

    Search author on:PubMed Google Scholar

  3. Stefan Krastanov
    View author publications

    Search author on:PubMed Google Scholar

  4. Ryan Hamerly
    View author publications

    Search author on:PubMed Google Scholar

  5. Nicholas Harris
    View author publications

    Search author on:PubMed Google Scholar

  6. Darius Bunandar
    View author publications

    Search author on:PubMed Google Scholar

  7. Matthew Streshinsky
    View author publications

    Search author on:PubMed Google Scholar

  8. Michael Hochberg
    View author publications

    Search author on:PubMed Google Scholar

  9. Dirk Englund
    View author publications

    Search author on:PubMed Google Scholar

Contributions

S.B. and D.E. conceived the experiments. S.B. designed the PIC, chip packaging and control electronics; calibrated the system; and conducted the experiments. A.S. assisted with characterizing the electro-optical nonlinearity. S.B., S.K. and D.E. developed the in situ training scheme. R.H. assisted with developing the calibration procedures for the system and interpreting the results of the in situ training experiments. N.H. and D.B. architected the PIC. D.B. performed the preliminary evaluation of the PIC in TensorFlow. M.S. and M.H. fabricated the PIC. S.B. and D.E. wrote the manuscript with input from all authors.

Corresponding authors

Correspondence to Saumil Bandyopadhyay or Dirk Englund.

Ethics declarations

Competing interests

S.B., R.H. and D.E. have filed US patent application nos. 17/556,033 and 17/711,640 on error correction algorithms for programmable photonics. A.S. is a photonics architect at Lightmatter and holds stock in the company. N.H. is the CEO of Lightmatter. D.B. is the Chief Scientist at Lightmatter. M.S. is the CEO of Enosemi. M.H. is the President of Periplous, LLC. D.E. holds shares in Lightmatter, but received no support for this work. The other authors declare no competing interests.

Peer review

Peer review information

Nature Photonics thanks Yichen Shen and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Figs. 1–13, Sections I–IX and Tables 1–3.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bandyopadhyay, S., Sludds, A., Krastanov, S. et al. Single-chip photonic deep neural network with forward-only training. Nat. Photon. 18, 1335–1343 (2024). https://doi.org/10.1038/s41566-024-01567-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue date:

  • DOI: https://doi.org/10.1038/s41566-024-01567-z

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing