Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Comment
  • Published:

Adaptive validation strategies for real-world clinical artificial intelligence

Nature Computational Science volume 5, pages 980–986 (2025)Cite this article

Subjects

Technical metrics used to evaluate medical artificial intelligence tools often fail to predict their clinical impact. We characterize this discordance and propose a framework of study designs to guide the translational process for clinical artificial intelligence tools, acknowledging their diversity and specific validation requirements.

This is a preview of subscription content, access via your institution

Relevant articles

Open Access articles citing this article.

Access options

Buy this article

  • Purchase on SpringerLink
  • Instant access to the full article PDF.

USD 39.95

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Conceptual differences in traditional study designs for retrospective development and randomized prospective clinical evaluation of computational tools in medicine.
Fig. 2: Study designs to generate real-world evidence from real-world data in clinical artificial intelligence.
Fig. 3: Translational framework for computational tools in medicine.
Fig. 4: Practical implementation scenarios of the proposed framework of clinical artificial intelligence study designs.

References

  1. Perez-Lopez, R., Ghaffari Laleh, N., Mahmood, F. & Kather, J. N. Nat. Rev. Cancer 24, 427–441 (2024).

    Article  Google Scholar 

  2. Reis-Filho, J. S. & Kather, J. N. J. Natl Cancer Inst. 115, 608–612 (2023).

    Article  Google Scholar 

  3. Daye, D. et al. Radiology 305, 555–563 (2022).

    Article  Google Scholar 

  4. Carstens, M. et al. Preprint at medRxiv https://doi.org/10.1101/2025.07.12.25330122 (2025).

  5. Maier-Hein, L. et al. Nat. Meth. https://doi.org/10.1038/s41592-023-02151-z (2024).

    Article  Google Scholar 

  6. Reinke, A. et al. Nat. Meth. https://doi.org/10.1038/s41592-023-02150-0 (2024).

    Article  Google Scholar 

  7. Yao, X. et al. Nat. Med. 27, 815–819 (2021).

    Article  Google Scholar 

  8. Djurisic, S. et al. Trials 18, 360 (2017).

    Article  Google Scholar 

  9. Faris, O. & Shuren, J. N. Engl. J. Med. 376, 1350–1357 (2017).

    Article  Google Scholar 

  10. Han, R. et al. Lancet Digital Health 6, e367–e373 (2024).

    Article  Google Scholar 

  11. Ford, I. & Norrie, J. N. Engl. J. Med. 375, 454–463 (2016).

    Article  Google Scholar 

  12. Castelo-Branco, L. et al. Ann. Oncol. 34, 1097–1112 (2023).

    Article  Google Scholar 

  13. González, J. et al. NEJM AI https://doi.org/10.1056/AIoa2400859 (2025).

  14. Feinberg, B. A. et al. Value Health 23, 1358–1365 (2020).

    Article  Google Scholar 

  15. Franklin, J. M., Glynn, R. J., Suissa, S. & Schneeweiss, S. Clin. Pharmacol. Ther. 107, 735–737 (2020).

    Article  Google Scholar 

  16. Mandrekar, S. J. & Sargent, D. J. J. Clin. Oncol. 27, 4027–4034 (2009).

    Article  Google Scholar 

  17. Kolbinger, F. R., Veldhuizen, G. P., Zhu, J., Truhn, D. & Kather, J. N. Commun. Med. 4, 71 (2024).

    Article  Google Scholar 

Download references

Acknowledgements

F.R.K. receives support from the German Cancer Research Center (CoBot 2.0), the Joachim Herz Foundation (Add-On Fellowship for Interdisciplinary Life Science), the Central Indiana Corporate Partnership AnalytiXIN Initiative, the Evan and Sue Ann Werling Pancreatic Cancer Research Fund, and the Indiana Clinical and Translational Sciences Institute (EPAR4157), funded, in part, by grant number UM1TR004402 from the National Institutes of Health, the National Center for Advancing Translational Sciences, Clinical and Translational Sciences Award. J.N.K. is supported by the German Cancer Aid DKH (DECADE, 70115166), the German Federal Ministry of Research, Technology and Space BMFTR (PEARL, 01KD2104C; CAMINO, 01EO2101; TRANSFORM LIVER, 031L0312A; TANGERINE, 01KT2302 through ERA-NET Transcan; Come2Data, 16DKZ2044A; DEEP-HCC, 031L0315A; DECIPHER-M, 01KD2420A; NextBIG, 01ZU2402A), the German Research Foundation DFG (CRC/TR 412, 535081457; SFB 1709/1 2025, 533056198), the German Academic Exchange Service DAAD (SECAI, 57616814), the German Federal Joint Committee G-BA (TransplantKI, 01VSF21048), the European Union’s Horizon Europe Research and Innovation Programme (ODELIA, 101057091; GENIAL, 101096312), the European Research Council ERC (NADIR, 101114631), the National Institutes of Health NIH (EPICO, R01 CA263318) and the National Institute for Health and Care Research NIHR (Leeds Biomedical Research Centre, NIHR203331). The views expressed are those of the authors and not necessarily those of the National Institutes of Health, the NHS, the NIHR or the Department of Health and Social Care. This work was funded by the European Union. Views and opinions expressed are, however, those of the authors only and do not necessarily reflect those of the European Union. Neither the European Union nor the granting authority can be held responsible for them.

Author information

Authors and Affiliations

  1. Weldon School of Biomedical Engineering, Purdue University, West Lafayette, IN, USA

    Fiona R. Kolbinger

  2. Department of Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany

    Fiona R. Kolbinger

  3. Else Kroener Fresenius Center for Digital Health, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany

    Jakob Nikolas Kather

  4. Department of Medicine I, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany

    Jakob Nikolas Kather

  5. Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany

    Jakob Nikolas Kather

  6. Pathology and Data Analytics, Leeds Institute of Medical Research at St James’s, University of Leeds, Leeds, UK

    Jakob Nikolas Kather

Authors
  1. Fiona R. Kolbinger
    View author publications

    Search author on:PubMed Google Scholar

  2. Jakob Nikolas Kather
    View author publications

    Search author on:PubMed Google Scholar

Contributions

Both authors developed the adaptive validation framework, wrote and revised the manuscript.

Corresponding authors

Correspondence to Fiona R. Kolbinger or Jakob Nikolas Kather.

Ethics declarations

Competing interests

F.R.K. declares advisory roles for Radical Health AI and the Surgical Data Science Collective. J.N.K. declares ongoing consulting services for AstraZeneca, Panakeia and Bioptimus. Furthermore, J.N.K. holds shares in Stratifai, Synagen and Spira Labs, has received an institutional research grant from GlaxoSmithKline, and has received honoraria from AstraZeneca, Bayer, Daiichi Sankyo, Eisai, Janssen, Merck, MSD, BMS, Roche, Pfizer and Fresenius.

Peer review

Peer review information

Nature Computational Science thanks the anonymous reviewer(s) for their contribution to the peer review of this work.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kolbinger, F.R., Kather, J.N. Adaptive validation strategies for real-world clinical artificial intelligence. Nat Comput Sci 5, 980–986 (2025). https://doi.org/10.1038/s43588-025-00901-x

Download citation

  • Published:

  • Version of record:

  • Issue date:

  • DOI: https://doi.org/10.1038/s43588-025-00901-x

This article is cited by

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics