Fig. 2

Workflow of the Multi-Modal Model (MuMo) for predicting treatment response to patients with anti-HER2 GC. a Feature extraction process: Pathological WSIs and radiological CT scans were processed to extract deep and omics features, which were correlated with clinical reports provided by pathologists or radiologists. b Multi-modal information fusion process: MuMo employs intra-modal fusion modules to integrate image features and clinical reports from pathology and radiology to obtain enhanced features. These features were then amalgamated using an inter-modal fusion module, and the patient information was incorporated using a separate patient information fusion module. Subsequently, a predictor was used to predict the risk scores. MuMo can handle missing modalities by employing learnable modality features as placeholders. c Overview of experimental pipeline: Data were sourced from Beijing Cancer Hospital (PKCancer) and external hospitals. The patients were divided into anti-HER2 and anti-HER2 combined immunotherapy, as well as external cohorts. The anti-HER2 cohort was randomly divided into a training set to train the model and a validation set to tune its parameters. The final model with frozen parameters was used to analyze the results. Additionally, an external cohort was used as an independent test set to test the robustness of the model. In the anti-HER2 combined immunotherapy cohort, a similar analytical process was employed