Abstract
Precise survival risk stratification for bladder urothelial carcinoma (BUC) remains a clinical challenge. We developed and validated a multimodal AI agent that integrates textual, radiographic, and pathological data from 1185 patients across four medical centers to predict survival risk. The agent employs LLMs to standardize pathology reports, interactive deep learning networks for precise CT image segmentation, and extracts features from CT scans and whole slide images using CTVisionNet and MacroVisionNet. The multimodal fusion framework, MATCH-Net, integrates these features with microscopic pathology information and clinical text embeddings using a multi-head attention mechanism to generate a comprehensive prognostic score. In multi-center validation, MATCH-Net demonstrated robust performance (C-index ranging from 0.836 to 0.874) and effectively stratified patients into high- and low-risk groups, identifying potential candidates responsive to adjuvant chemotherapy. Furthermore, the framework enabled the quantification of novel, interpretable prognostic biomarkers and provides a reliable and clinically applicable solution for personalized BUC prognosis.
Similar content being viewed by others
Data availability
The WSIs, nephrographic CT scans, and annotation data used for both the training and validation sets are subject to institutional restrictions. Due to patient privacy obligations and Institutional Review Board (IRB) approvals, these data are not publicly available. However, they can be accessed upon reasonable request from the corresponding author, pending approval from the IRBs and legal departments of all participating centers.
Code availability
The source code is available online (https://github.com/hqh1997/MMS_AI_agent).
References
Nadal, R., Valderrama, B. P. & Bellmunt, J. Progress in systemic therapy for advanced-stage urothelial carcinoma. Nat. Rev. Clin. Oncol. 21, 8–27 (2024).
Hemenway, G. et al. Advancements in Urothelial Cancer Care: Optimizing Treatment for Your Patient. Am. Soc. Clin. Oncol. Educ. Book 44, e432054 (2024).
Lopez-Beltran, A., Cookson, M. S., Guercio, B. J. & Cheng, L. Advances in diagnosis and treatment of bladder cancer. BMJ 384, e076743 (2024).
Soualhi, A. et al. The incidence and prevalence of upper tract urothelial carcinoma: a systematic review. BMC Urol. 21, 110 (2021).
Compérat, E. et al. Current best practice for bladder cancer: a narrative review of diagnostics and treatments. Lancet 400, 1712–1721 (2022).
Shen, J., Li, Z., Wang, R., Ding, G. & Zhang, Y. Bladder cancer diagnostic and prognostic models from DNA methylation by multi algorithm machine learning. NPJ Precis. Oncol. https://doi.org/10.1038/s41698-025-01195-y (2025).
Raman, S. P. & Fishman, E. K. Bladder malignancies on CT: the underrated role of CT in diagnosis. AJR Am. J. Roentgenol. 203, 347–354 (2014).
Compérat, E. et al. Updated pathology reporting standards for bladder cancer: biopsies, transurethral resections and radical cystectomies. World J. Urol. 40, 915–927 (2022).
Perez-Lopez, R., Ghaffari Laleh, N., Mahmood, F. & Kather, J. N. A guide to artificial intelligence for cancer researchers. Nat. Rev. Cancer https://doi.org/10.1038/s41568-024-00694-7 (2024).
McGenity, C. et al. Artificial intelligence in digital pathology: a systematic review and meta-analysis of diagnostic test accuracy. npj Dig. Med. 7, 114 (2024).
He, J. et al. Development of a deep learning model for T1N0 gastric cancer diagnosis using 2.5D radiomic data in preoperative CT images. NPJ Precis. Oncol. 9, 249 (2025).
Isensee, F., Jaeger, P. F., Kohl, S. A. A., Petersen, J. & Maier-Hein, K. H. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nat. Methods 18, 203–211 (2021).
Çiçek, Ö., Abdulkadir, A., Lienkamp, S. S., Brox, T. & Ronneberger, O. in Medical Image Computing and Computer-Assisted Intervention – MICCAI 2016 (eds Sebastien Ourselin et al) 424–432 (Springer International Publishing, 2016).
Shao, Z. TransMIL: transformer based correlated multiple instance learning for whole slide image classification. In Proc. 35th International Conference on Neural Information Processing Systems 2136–2147 (ACM, 2021).
Chen, R. J. et al. in Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part VIII 24. 339–349 (Springer, 2021).
Saldanha, O. L. et al. Self-supervised attention-based deep learning for pan-cancer mutation prediction from histopathology. NPJ Precis. Oncol. 7, 35 (2023).
Lu, M. Y. et al. A visual-language foundation model for computational pathology. Nat. Med. 30, 863–874 (2024).
Chen, R. J. et al. Towards a general-purpose foundation model for computational pathology. Nat. Med. 30, 850–862 (2024).
Yuan, Y. et al. Cell graph analysis in hepatocellular carcinoma: predicting local recurrence and identifying spatial relationship biomarkers. NPJ Precis. Oncol. 9, 261 (2025).
Pisula, J. I. et al. Explainable, federated deep learning model predicts disease progression risk of cutaneous squamous cell carcinoma. NPJ Precis. Oncol. 9, 205 (2025).
Lu, M. Y. et al. Data-efficient and weakly supervised computational pathology on whole-slide images. Nat. Biomed. Eng. 5, 555–570 (2021).
Chen, R. J. et al. Pan-cancer integrative histology-genomic analysis via multimodal deep learning. Cancer Cell 40, 865–878.e866 (2022).
Wang, G. et al. DeepIGeoS: a deep interactive geodesic framework for medical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 41, 1559–1572 (2018).
Luo, X. et al. MIDeepSeg: minimally interactive segmentation of unseen objects from medical images using deep learning. Med. Image Anal. 72, 102102 (2021).
Liang, J. et al. Deep learning supported discovery of biomarkers for clinical prognosis of liver cancer. Nat. Mach. Intell. 5, 408–420 (2023).
He, Q. et al. Integrated multicenter deep learning system for prognostic prediction in bladder cancer. NPJ Precis. Oncol. 8, 233 (2024).
Lee, Y., Ferber, D., Rood, J. E., Regev, A. & Kather, J. N. How AI agents will change cancer research and oncology. Nat. Cancer 5, 1765–1767 (2024).
Ferber, D. et al. Development and validation of an autonomous artificial intelligence agent for clinical decision-making in oncology. Nat. Cancer https://doi.org/10.1038/s43018-025-00991-6 (2025).
Liu, X. et al. A generalist medical language model for disease diagnosis assistance. Nat. Med. 31, 932–942 (2019).
Selvaraju, R. R. et al. Grad-CAM: visual explanations from deep networks via gradient-based localization. Int. J. Comput. Vision 128, 336–359 (2020).
Suzuki, H. et al. Exploring drug resistance via intercellular crosstalk using spatial transcriptomics in high-grade serous ovarian carcinoma. NPJ Precis. Oncol. 9, 345 (2025).
Li, T. et al. Computational pathology annotation enhances the resolution and interpretation of breast cancer spatial transcriptomics data. NPJ Precis. Oncol. 9, 310 (2025).
Hatamizadeh, A. et al. in International MICCAI brainlesion workshop. 272–284 (Springer).
Kirillov, A. et al. Segment anything. in Proc. IEEE/CVF International Conference on Computer Vision 4015–4026 (IEEE, 2023).
Xie, S., Girshick, R., Dollár, P., Tu, Z. & He, K. Aggregated residual transformations for deep neural networks. in Proc. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 5987–5995 (IEEE, 2017).
Ganesan, K. ROUGE 2.0: Updated and Improved Measures for Evaluation of Summarization Tasks. Preprint at https://arxiv.org/abs/1803.01937 (2018).
Zhang, T. et al. BERTScore: Evaluating text generation with BERT. Int. Conf. Learn. Represent. (2020).
Chen, R. J. et al. in Proc. IEEE/CVF International Conference on Computer Vision (ICCV) 3995–4005 (IEEE, 2021).
Chen, R. J. et al. Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis. IEEE Trans. Med. Imaging 41, 757–770 (2022).
Kokhlikyan, N. et al. Captum: a unified and generic model interpretability library for PyTorch. Preprint at https://doi.org/10.48550/arXiv.2009.07896 (2020).
Zadeh, S. G. & Schmid, M. Bias in cross-entropy-based training of deep survival networks. IEEE Trans. Pattern Anal. Mach. Intell. 43, 3126–3137 (2021).
Acknowledgements
We acknowledge the support from the Medical Health Care Ecosystem Innovation Team of the First Affiliated Hospital of Chongqing Medical University (CYYY-DSTDXM-202409), the Postgraduate Education Reform Project of the First Affiliated Hospital of Chongqing Medical University (jgxm-202501), and the Chongqing Municipal Education Commission's 14th 5-year key discipline Support Project (No. 20240101). We thank all pathologists, radiologists, and related staff at the participating institutions for their assistance in data collection. Computing work was partly supported by the Supercomputing Center of Chongqing Medical University.
Author information
Authors and Affiliations
Contributions
Q.H.H., H.T., B.X.X., Y.W.T., X.P., W.Y.H., and M.Z.X. conceived and designed the study; H.T., C.J.P., X.F.Y., X.P., and X.Z. collected the data. Q.H.H., H.T., and C.J.P. evaluated images. Y.J.L., Y.W.T., and D.Y.C. labeled the pathological slide images. F.J.L. supervised and annotated the radiographic images. Q.H.H., W.L.Z., and X.P. trained and developed the AI system. Q.H.H., B.X.X., and Y.W.T. analyzed and interpreted the data and wrote the original draft of the manuscript. Q.H.H. and X.F.Y. were responsible for revising the manuscript and performing supplementary experiments. W.L.Z., X.F.Y., W.Y.H., and M.Z.X. supervised and directed the study.
Corresponding authors
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
About this article
Cite this article
He, Q., Tan, H., Xiao, B. et al. Development and validation of a multimodal AI-agent system for prognosis analysis of bladder urothelial carcinoma. npj Precis. Onc. (2026). https://doi.org/10.1038/s41698-026-01415-z
Received:
Accepted:
Published:
DOI: https://doi.org/10.1038/s41698-026-01415-z
