Fig. 2: Workflow of the study.

a Model development. We developed and evaluated a deep learning system to predict the treatment response of breast cancer patients across neoadjuvant therapy (NAT). The system incorporates deep neural networks trained on Pre-NAT mammogram images and longitudinal MRI scans, along with rhpc information (radiological assessments, histopathological assessments, personal patient records, and clinical data). After data retrieval, iMGrhpc and iMRrhpc were modeled independently, where iMGrhpc is based on Pre-NAT mammogram and rhpc data, while iMRrhpc is based on longitudinal MRIs embedding temporal information and rhpc data. Both models include two modules: one module is for cross-modal knowledge learning that predicts rhpc information using only imaging features, and another module is for response prediction using integrated features of rhpc-based and imaging features. These models were further combined into the Multi-modal Response Prediction (MRP) system. MLP refers to a two-layer multi-layer perceptron with an output dimension of 256. b Datasets. The internal dataset was collected from the Netherlands Cancer Institute and was randomly partitioned into training, validation, and test subsets. For evaluating our system on unseen data, we collected three external datasets from different centers: Duke University (United States; n = 288), Fuzhou Province Hospital (China; n = 85), and I-SPY2 (United States; n = 508). c NAT response assessment of AI model and reader study. We assessed MRP's ability to predict pathological response (pCR vs. non-pCR) at different stages-Pre-NAT (before administration of NAT), Mid-NAT (during therapy), and Post-NAT (prior to surgery)-using standard metrics: AUROC (Area Under Receiver Operating Characteristic Curve) and AUPRC (Area Under Precision-Recall Curve). To compare the performance of MRP with human experts, we conducted a reader study involving six international breast radiologists. The average performance of the readers is indicated with a red “+R" in the plot. d Personalizing management in clinical practice. We simulated two scenarios to assess the system’s ability to personalize treatment: identifying non-pCR patients before NAT in whom toxic treatments may be timely adapted, and identifying pCR patients before surgery for the potential reduction of surgical procedures. Circled C indicates current clinical practice; Circled AI indicates our MRP system suggested strategy.