Fig. 5

Interpretability analysis of the Multi-Modal Model (MuMo). a Visualization of the importance scores of ābagsā on pathological whole-slide images. Darker red regions signify a higher contribution to the response prediction, whereas darker blue regions suggest a diminished influence. The second row shows the four most important bags on the slide image. b Visualization of attention maps on radiological lesion images using the Grad-CAM algorithm. Darker red regions signify heightened attention from MuMo, whereas darker blue regions denote reduced attention. The red bounding box emphasizes MuMoās predominant focus on lymph node and liver tumors in this responder. cāg Evaluation of predicted risk scores across various clinical information subgroups in the anti-HER2 cohort. hāl Evaluation of predicted risk scores across various clinical information subgroups in the anti-HER2 combined immunotherapy cohort