Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Deep-learning-based HER2 status assessment from multimodal breast cancer data predicts neoadjuvant therapy response

Nature Biomedical Engineering (2025)Cite this article

Subjects

Abstract

Accurate assessment of human epidermal growth factor receptor 2 (HER2) status is crucial for effective breast cancer treatment planning and improved patient outcomes. Traditional needle biopsies, limited in tissue sampling, often lead to inaccurate assessments due to intratumoural heterogeneity. Here, to address this, we introduce the deep-learning-based HER2 multimodal alignment and prediction (MAP) model, which leverages pretreatment multimodal breast cancer images for a more comprehensive reflection of tumour characteristics and provides more accurate HER2 status prediction. We develop patient response MAP models to demonstrate the HER2 prediction performance of our model compared with needle biopsies from patients receiving neoadjuvant therapy. A large-scale multimodal breast cancer dataset from 4 centres, consisting of 14,472 images from 6,991 cases, is adopted in this study, and the results consistently demonstrate the superiority of our HER2 MAP model in predicting patient response. These findings highlight the substantial advantages of our HER2 predictions. Our study provides physicians with a crucial tool for informed clinical decisions and treatment plans, aiming to improve outcomes in patients with breast cancer.

Access through your institution
Buy or subscribe

This is a preview of subscription content, access via your institution

Access options

Access through your institution

Buy this article

  • Purchase on SpringerLink
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Overview of motivation, experimental data and method design in the study.
Fig. 2: Deep-learning-based prediction workflows and models.
Fig. 3: HER2 status prediction analysis.
Fig. 4: HER2-response prediction performance.
Fig. 5: Element contribution analysis.

Data availability

We possess multimodal datasets collected from four medical centres. The images and clinical information of centre C are available at https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=70226903. Requests for academic use of in-house raw data can be addressed to the corresponding authors. All requests will be promptly reviewed to assess any intellectual property or patient confidentiality obligations, and will be processed in accordance with institutional and departmental guidelines, subject to a material transfer agreement. Source data are provided with this paper.

Code availability

Code for PyTorch implementation of MAP is available via GitHub at https://github.com/ZhangJD-ong/HER2-MAP-from-Multimodal-Breast-Data (ref. 46).

References

  1. Siegel, R. L., Miller, K. D., Wagle, N. S. & Jemal, A. Cancer statistics, 2023. CA Cancer J. Clin. 73, 17–48 (2023).

    PubMed  Google Scholar 

  2. Siegel, R. L., Giaquinto, A. N. & Jemal, A. Cancer statistics, 2024. CA Cancer J. Clin. 74, 12–49 (2024).

    PubMed  Google Scholar 

  3. Giaquinto, A. N. et al. Breast cancer statistics, 2022. CA Cancer J. Clin. 72, 524–541 (2022).

    PubMed  Google Scholar 

  4. Zhang, J. et al. A robust and efficient AI assistant for breast tumor segmentation from DCE-MRI via a spatial–temporal framework. Patterns 4, 100826 (2023).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  5. Park, Y. H. et al. Chemotherapy induces dynamic immune responses in breast cancers that impact treatment outcome. Nat. Commun. 11, 6175 (2020).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  6. Bianchini, G., De Angelis, C., Licata, L. & Gianni, L. Treatment landscape of triple-negative breast cancer—expanded options, evolving needs. Nat. Rev. Clin. Oncol. 19, 91–113 (2022).

    Article  CAS  PubMed  Google Scholar 

  7. Asselain, B. et al. Long-term outcomes for neoadjuvant versus adjuvant chemotherapy in early breast cancer: meta-analysis of individual patient data from ten randomised trials. Lancet Oncol. 19, 27–39 (2018).

    Article  Google Scholar 

  8. Agostinetto, E., Gligorov, J. & Piccart, M. Systemic therapy for early-stage breast cancer: learning from the past to build the future. Nat. Rev. Clin. Oncol. 19, 763–774 (2022).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  9. McNamara, K. L. et al. Spatial proteomic characterization of HER2-positive breast tumors through neoadjuvant therapy predicts response. Nat. Cancer 2, 400–413 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  10. Cardoso, F. et al. Early breast cancer: ESMO clinical practice guidelines for diagnosis, treatment and follow-up. Ann. Oncol. 30, 1194–1220 (2019).

    Article  CAS  PubMed  Google Scholar 

  11. Korde, L. A. et al. Neoadjuvant chemotherapy, endocrine therapy, and targeted therapy for breast cancer: ASCO guideline. J. Clin. Oncol. 39, 1485 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  12. Wiklander, O. P. et al. Antibody-displaying extracellular vesicles for targeted cancer therapy. Nat. Biomed. Eng. 8, 1453–1468 (2024).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  13. Von Minckwitz, G. et al. Adjuvant pertuzumab and trastuzumab in early HER2-positive breast cancer. N. Engl. J. Med. 377, 122–131 (2017).

    Article  Google Scholar 

  14. Gianni, L. et al. 5-year analysis of neoadjuvant pertuzumab and trastuzumab in patients with locally advanced, inflammatory, or early-stage HER2-positive breast cancer (NeoSphere): a multicentre, open-label, phase 2 randomised trial. Lancet Oncol. 17, 791–800 (2016).

    Article  CAS  PubMed  Google Scholar 

  15. Rakha, E. A. & Ellis, I. O. Updated guideline recommendations for HER2 testing. Nat. Rev. Clin. Oncol. 11, 8–9 (2014).

    Article  PubMed  Google Scholar 

  16. Boba, M. et al. False-negative results of breast core needle biopsies—retrospective analysis of 988 biopsies. Pol. J. Radiol. 76, 25 (2011).

    PubMed  PubMed Central  Google Scholar 

  17. Filho, O. M. et al. Impact of HER2 heterogeneity on treatment response of early-stage HER2-positive breast cancer: phase II neoadjuvant clinical trial of T-DM1 combined with pertuzumab. Cancer Discov. 11, 2474–2487 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  18. Su, G.-H. et al. Radiogenomic-based multiomic analysis reveals imaging intratumor heterogeneity phenotypes and therapeutic targets. Sci. Adv. 9, eadf0837 (2023).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  19. Cahill, R. A., Walsh, D., Landers, R. J. & Watson, R. G. Preoperative profiling of symptomatic breast cancer by diagnostic core biopsy. Ann. Surg. Oncol. 13, 45–51 (2006).

    Article  PubMed  Google Scholar 

  20. Ough, M., Velasco, J. & Hieken, T. J. A comparative analysis of core needle biopsy and final excision for breast cancer: histology and marker expression. Am. J. Surg. 201, 692–694 (2011).

    Article  PubMed  Google Scholar 

  21. Lorgis, V. et al. Discordance in early breast cancer for tumour grade, estrogen receptor, progesteron receptors and human epidermal receptor-2 status between core needle biopsy and surgical excisional primary tumour. Breast 20, 284–287 (2011).

  22. Chen, J. et al. Comparison of core needle biopsy and excision specimens for the accurate evaluation of breast cancer molecular markers: a report of 1003 cases. Pathol. Oncol. Res. 23, 769–775 (2017).

    Article  CAS  PubMed  Google Scholar 

  23. Shah, V. I. et al. False-negative core needle biopsies of the breast: an analysis of clinical, radiologic, and pathologic findings in 27 consecutive cases of missed breast cancer. Cancer 97, 1824–1831 (2003).

    Article  PubMed  Google Scholar 

  24. Zhang, J., Wu, J., Zhou, X. S., Shi, F. & Shen, D. Recent advancements in artificial intelligence for breast cancer: image augmentation, segmentation, diagnosis, and prognosis approaches. Semin. Cancer Biol. 96, 11–25 (2023).

    Article  CAS  PubMed  Google Scholar 

  25. Nicosia, L. et al. Radiomic features applied to contrast enhancement spectral mammography: possibility to predict breast cancer molecular subtypes in a non-invasive manner. Int. J. Mol. Sci. 23, 15322 (2022).

    Article  PubMed  PubMed Central  Google Scholar 

  26. Cui, H. et al. Radiogenomic analysis of prediction HER2 status in breast cancer by linking ultrasound radiomic feature module with biological functions. J. Transl. Med. 21, 44 (2023).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  27. Ramtohul, T. et al. Multiparametric mri and radiomics for the prediction of HER2-zero, -low, and-positive breast cancers. Radiology 308, e222646 (2023).

    Article  PubMed  Google Scholar 

  28. Feng, L. et al. Development and validation of a radiopathomics model to predict pathological complete response to neoadjuvant chemoradiotherapy in locally advanced rectal cancer: a multicentre observational study. Lancet Digit. Health 4, e8–e17 (2022).

    Article  CAS  PubMed  Google Scholar 

  29. Zhou, H.-Y. et al. A transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics. Nat. Biomed. Eng. 7, 743–755 (2023).

    Article  PubMed  Google Scholar 

  30. Acosta, J. N., Falcone, G. J., Rajpurkar, P. & Topol, E. J. Multimodal biomedical AI. Nat. Med. 28, 1773–1784 (2022).

    Article  CAS  PubMed  Google Scholar 

  31. Qian, X. et al. Prospective assessment of breast cancer risk from multimodal multiview ultrasound images via clinically applicable deep learning. Nat. Biomed. Eng. 5, 522–532 (2021).

    Article  PubMed  Google Scholar 

  32. Boehm, K. M., Khosravi, P., Vanguri, R., Gao, J. & Shah, S. P. Harnessing multimodal data integration to advance precision oncology. Nat. Rev. Cancer 22, 114–126 (2022).

    Article  CAS  PubMed  Google Scholar 

  33. Li, Z. et al. Domain generalization for mammography detection via multi-style and multi-view contrastive learning. In Proc. 24th International Conference on Medical Image Computing and Computer-Assisted Intervention–MICCAI 2021 (eds de Bruijne, M. et al.) 98–108 (Springer, 2021).

  34. Tian, Z., Shen, C., Chen, H. & He, T. FCOS: a simple and strong anchor-free object detector. IEEE Trans. Pattern Anal. Mach. Intell. 44, 1922–1933 (2020).

    Google Scholar 

  35. Zhou, B., Khosla, A., Lapedriza, A., Oliva, A. & Torralba, A. Learning deep features for discriminative localization. In Proc. IEEE Conference on Computer Vision and Pattern Recognition 2921–2929 (IEEE, 2016).

  36. Lu, W. et al. Slidegraph+: whole slide image level graphs to predict HER2 status in breast cancer. Med. Image Anal. 80, 102486 (2022).

    Article  PubMed  Google Scholar 

  37. La Forgia, D. et al. Radiomic analysis in contrast-enhanced spectral mammography for predicting breast cancer histological outcome. Diagnostics 10, 708 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  38. Yuan, Y. et al. Accuracy of MRI in prediction of pathologic complete remission in breast cancer after preoperative therapy: a meta-analysis. Am. J. Roentgenol. 195, 260–268 (2010).

    Article  Google Scholar 

  39. Baumgartner, A. et al. Ultrasound-based prediction of pathologic response to neoadjuvant chemotherapy in breast cancer patients. Breast 39, 19–23 (2018).

    Article  PubMed  Google Scholar 

  40. Weber, J. J. et al. MRI and prediction of pathologic complete response in the breast and axilla after neoadjuvant chemotherapy for breast cancer. J. Am. Coll. Surg. 225, 740–746 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  41. Bitencourt, A. G. et al. MRI-based machine learning radiomics can predict HER2 expression level and pathologic response after neoadjuvant therapy in HER2 overexpressing breast cancer. eBioMedicine 61, 103042 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  42. Turnbull, L. W. Dynamic contrast-enhanced MRI in the diagnosis and management of breast cancer. NMR Biomed. 22, 28–39 (2009).

  43. Li, M. et al. Developing large pre-trained model for breast tumor segmentation from ultrasound images. In Proc. International Conference on Medical Image Computing and Computer-Assisted Intervention (eds Greenspan, H. et al.) 89–96 (Springer, 2023).

  44. Mo, Y. et al. Hover-Trans: anatomy-aware Hover-Transformer for ROI-free breast cancer diagnosis in ultrasound images. IEEE Trans. Med. Imag. 42, 1696–1706 (2023).

    Article  Google Scholar 

  45. Liu, Z. et al. Swin Transformer: hierarchical vision transformer using shifted windows. In Proc. IEEE/CVF International Conference on Computer Vision 10012–10022 (IEEE, 2021).

  46. Zhang, J. HER2 MAP from multimodal breast data. GitHub https://github.com/ZhangJD-ong/HER2-MAP-from-Multimodal-Breast-Data (2025).

Download references

Acknowledgements

This work was supported in part by National Natural Science Foundation of China (grant numbers U23A20295 (D.S.), 82441023 (D.S.), 62131015 (D.S.), 82001986 (Zhenhui Li), 82360345 (Zhenhui Li), 82202143 (Zhenhui Li), 62250710165 (Zhenhui Li)), the China Ministry of Science and Technology (S20240085, STI2030-Major Projects-2022ZD0209000, STI2030-Major Projects-2022ZD0213100) (D.S.), Shanghai Municipal Central Guided Local Science and Technology Development Fund (grant number YDZX20233100001001) (D.S.), the Innovative Research Team of Yunnan Province (grant number 202505AS350013, Z.L.) and HPC Platform of ShanghaiTech University.

Author information

Author notes

  1. These authors contributed equally: Jiadong Zhang, Yonghao Li, Zheren Li.

Authors and Affiliations

  1. School of Biomedical Engineering and State Key Laboratory of Advanced Medical Materials and Devices, ShanghaiTech University, Shanghai, China

    Jiadong Zhang, Yonghao Li, Zhiming Cui & Dinggang Shen

  2. Department of Biomedical Engineering, City University of Hong Kong, Hong Kong, China

    Jiadong Zhang, Kannie W. Y. Chan & Qinrong Zhang

  3. Hong Kong Centre for Cerebro-cardiovascular Health Engineering, Hong Kong, China

    Jiadong Zhang & Kannie W. Y. Chan

  4. School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, China

    Zheren Li

  5. Shanghai United Imaging Intelligence Co. Ltd., Shanghai, China

    Zheren Li & Dinggang Shen

  6. Department of Radiology, The Affiliated Hospital of Southwest Medical University, Luzhou, China

    Pinxiong Li

  7. Department of Radiology, The First Affiliated Hospital of Kunming Medical University, Kunming, China

    Jun Li

  8. Third Department of Breast Surgery, The Third Affiliated Hospital of Kunming Medical University, Yunnan Cancer Hospital, Kunming, China

    Zhen Li

  9. Department of Radiology, The Third Affiliated Hospital of Kunming Medical University, Yunnan Cancer Hospital, Kunming, China

    Yu Xie & Zhenhui Li

  10. Tung Biomedical Sciences Centre, City University of Hong Kong, Hong Kong, China

    Qinrong Zhang

  11. Shanghai Clinical Research and Trial Center, Shanghai, China

    Dinggang Shen

Authors
  1. Jiadong Zhang
    View author publications

    Search author on:PubMed Google Scholar

  2. Yonghao Li
    View author publications

    Search author on:PubMed Google Scholar

  3. Zheren Li
    View author publications

    Search author on:PubMed Google Scholar

  4. Zhiming Cui
    View author publications

    Search author on:PubMed Google Scholar

  5. Pinxiong Li
    View author publications

    Search author on:PubMed Google Scholar

  6. Jun Li
    View author publications

    Search author on:PubMed Google Scholar

  7. Zhen Li
    View author publications

    Search author on:PubMed Google Scholar

  8. Yu Xie
    View author publications

    Search author on:PubMed Google Scholar

  9. Kannie W. Y. Chan
    View author publications

    Search author on:PubMed Google Scholar

  10. Qinrong Zhang
    View author publications

    Search author on:PubMed Google Scholar

  11. Zhenhui Li
    View author publications

    Search author on:PubMed Google Scholar

  12. Dinggang Shen
    View author publications

    Search author on:PubMed Google Scholar

Contributions

In this study, J.Z., Zhenhui Li, Q.Z. and D.S. designed the method and drafted the paper. J.Z. and Y.L. wrote the code. J.Z., Y.L., P.L., J.L., Zheren Li and Zhen Li collected and processed the dataset. J.Z. and Y.L. provided statistical analysis and interpretation of the data. Z.C., K.W.Y.C., Zhenhui Li and D.S. coordinated and supervised the whole work. All authors were involved in critical revisions of the paper and have read and approved the final version.

Corresponding authors

Correspondence to Kannie W. Y. Chan, Qinrong Zhang, Zhenhui Li or Dinggang Shen.

Ethics declarations

Competing interests

D.S. and Z.L. are consultant and employee of Shanghai United Imaging Intelligence Co., Ltd. The company had no role in designing or performing the surveillance, nor in analysing or interpreting the data. The other authors declare no competing interests.

Peer review

Peer review information

Nature Biomedical Engineering thanks the anonymous reviewers for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary notes, Figs. 1 and 2, and Tables 1–4.

Reporting Summary

Source data

Source Data Figs. 3–5

Figure 3a. HER2 status from needle biopsy, surgical biopsy and prediction. Figure 3b–f. Summary of HER2 prediction results. Figure 4a–d. HER2 status from needle biopsy and prediction. Figure 4e,f. Summary of therapy response prediction results. Figure 4k–r. AUROC of therapy response prediction results. Figure 5a. Percentage of contributions for each clinical feature. Figure 5b. Summary of therapy response prediction results of different combinations of modalities

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, J., Li, Y., Li, Z. et al. Deep-learning-based HER2 status assessment from multimodal breast cancer data predicts neoadjuvant therapy response. Nat. Biomed. Eng (2025). https://doi.org/10.1038/s41551-025-01495-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1038/s41551-025-01495-5

Search

Advanced search

Quick links

Nature Briefing: Cancer

Sign up for the Nature Briefing: Cancer newsletter — what matters in cancer research, free to your inbox weekly.

Get what matters in cancer research, free to your inbox weekly. Sign up for Nature Briefing: Cancer