Abstract
Patient safety and high treatment quality are essential in modern healthcare, but analyzing safety incidents for insights is labor-intensive and inconsistent process. To address this, we developed the Artificial Intelligence-based Incident Analysis and Learning System (AI-ILS), trained on 1548 expertly curated incidents categorized by the Human Factors Analysis and Classification System (HFACS). AI-ILS identifies latent safety threats and classifies incident causes with high accuracy, achieving an average AUROC of 0.92, MCC of 0.72, and overall accuracy of 79%. In testing on 350 real-world clinical incidents, AI-ILS showed 88% concordance with expert reviewers and operated 29 times faster than manual analysis. We deployed and validated AI-ILS using real-world radiation oncology data, where it improved retrospective incident analysis at our institution by generating aggregated HFACS-based results and addressing challenges related to inconsistent review processes and lack of standardized taxonomies.
Similar content being viewed by others
Data availability
Requests for raw and analyzed data will undergo internal review by MSKCC to assess whether they are subject to intellectual property rights or confidentiality restrictions. Any data deemed shareable will be provided under a material transfer agreement for non-commercial research purposes.Requests for programming code, fine-tuned models, and related materials will undergo internal review by MSKCC to assess whether they are subject to intellectual property rights or confidentiality restrictions. Any materials deemed shareable will be provided under a material transfer agreement for non-commercial research purposes.
References
Patient Safety: Achieving a New Standard for Care. (National Academies Press, Washington, D.C., 2004). https://doi.org/10.17226/10863.
Lee, S. E. et al. Safety culture, patient safety, and quality of care outcomes: A literature review. West J. Nurs. Res. 41, 279–304 (2019).
DiCuccio, M. H. The relationship between patient safety culture and patient outcomes: A systematic review. J. Patient Saf. 11, 135–142 (2015).
Institute of Medicine (US) Committee on Quality of Health Care in America. To Err Is Human: Building a Safer Health System. (National Academies Press (US), Washington (DC), 2000).
Zarei, M., Gershan, V. & Holmberg, O. Safety in radiation oncology (SAFRON): Learning about incident causes and safety barriers in external beam radiotherapy. Phys. Med. 111, 102618 (2023).
Nelson, C., Roy, L. A. & Wallace, H. J. Radiation oncology incident learning system (RO-ILS): Increasing stakeholder participation for safety and quality improvement. J. Clin. Oncol. 37, 232–232 (2019).
Milosevic, M. et al. The Canadian National System for Incident Reporting in Radiation Treatment (NSIR-RT) taxonomy. Pr. Radiat. Oncol. 6, 334–341 (2016).
Ford, E. C. & Evans, S. B. Incident learning in radiation oncology: A review. Med Phys. 45, e100–e119 (2018).
Ford, E. C., Fong de Los Santos, L., Pawlicki, T., Sutlief, S. & Dunscombe, P. Consensus recommendations for incident learning database structures in radiation oncology. Med Phys. 39, 7272–7290 (2012).
Thomadsen, B. et al. AAPM task group report 288: Recommendations for guiding radiotherapy event narratives. Med Phys. 51, 5858–5872 (2024).
Huq, M. S. et al. The report of Task Group 100 of the AAPM: Application of risk analysis methods to radiation therapy quality management. Med. Phys. 43, 4209–4262 (2016).
Yang, F. et al. Validating FMEA output against incident learning data: A study in stereotactic body radiation therapy. Med. Phys. 42, 2777–2785 (2015).
Paradis, K. C. et al. The fusion of incident learning and failure mode and effects analysis for data-driven patient safety improvements. Pr. Radiat. Oncol. 11, e106–e113 (2021).
Shappell, S. A. & Wiegmann, D. A. The Human Factors Analysis and Classification System—HFACS. (2000).
Jalali, M., Dehghan, H., Habibi, E. & Khakzad, N. Application of ‘Human Factor Analysis and Classification System’ (HFACS) model to the prevention of medical errors and adverse events: A systematic review. Int J. Prev. Med. 14, 127 (2023).
Diller, T. et al. The human factors analysis classification system (HFACS) applied to health care. Am. J. Med. Qual. 29, 181–190 (2014).
Lee, J.-Y. et al. Applying the human factors analysis and classification system (HFACS) within root cause analysis (RCA) to prevent medical errors and enhancing patient safety culture: Insights from a medical center. Int. J. Quality Health Care https://doi.org/10.1093/intqhc/mzaf009 10.1093/intqhc/mzaf009. (2025).
Mosaly, P. R. et al. Application of human factors analysis and classification system model to event analysis in radiation oncology. Pract. Radiat. Oncol. 5, 113–119 (2015).
Chan, A. J. et al. The use of human factors methods to identify and mitigate safety issues in radiation therapy. Radiother. Oncol. 97, 596–600 (2010).
Judy, G. D. et al. Incorporating human factors analysis and classification system (HFACS) into analysis of reported near misses and incidents in radiation oncology. Pr. Radiat. Oncol. 10, e312–e321 (2020).
McGurk, R. et al. Multi-institutional stereotactic body radiation therapy incident learning: Evaluation of safety barriers using a human factors analysis and classification system. J. Patient Saf. 19, e18–e24 (2023).
Weintraub, S. M., Salter, B. J., Chevalier, C. L. & Ransdell, S. Human factor associations with safety events in radiation therapy. J. Appl Clin. Med Phys. 22, 288–294 (2021).
RO-ILS and Clarity PSO. Aggregate Data Report Quarter 3, 2024. https://www.astro.org/getmedia/49e5d239-14e3-4e93-b148-75b9f3ce178e/ROILS_2024_Q3.pdf (2024).
Savova, G. K. et al. Mayo clinical text analysis and knowledge extraction system (cTAKES): Architecture, component evaluation and applications. J. Am. Med. Inform. Assoc. 17, 507–513 (2010).
Yalamanchili, A. et al. Quality of large language model responses to radiation oncology patient care questions. JAMA Netw. Open 7, e244630 (2024).
Wang, P. et al. Fine-tuning large language models for radiation oncology, a specialized health care domain. Int. J. Radiat. Oncol. *Biol. *Phys. 120, e664 (2024).
Ganguly, I., Buhrman, G., Kline, E., Mun, S. K. & Sengupta, S. Automated error labeling in radiation oncology via statistical natural language processing. Diagnostics 13, 1215 (2023).
Mathew, F., Wang, H., Montgomery, L. & Kildea, J. Natural language processing and machine learning to assist radiation oncology incident learning. J. Appl Clin. Med Phys. 22, 172–184 (2021).
Syed, K. et al. Automatic incident triage in radiation oncology incident learning system. Healthcare 8, 272 (2020).
Chen, H., Cohen, E., Wilson, D. & Alfred, M. A machine learning approach with human-AI collaboration for automated classification of patient safety event reports: Algorithm development and validation study. JMIR Hum. Factors 11, e53378 (2024).
Tabaie, A., Sengupta, S., Pruitt, Z. M. & Fong, A. A natural language processing approach to categorise contributing factors from patient safety event reports. BMJ Health Care Inf. 30, e100731 (2023).
Fong, A. Realizing the power of text mining and natural language processing for analyzing patient safety event narratives: The challenges and path forward. J. Patient Saf. 17, e834–e836 (2021).
Aronson, A. R. Effective mapping of biomedical text to the UMLS Metathesaurus: the MetaMap program. Proc AMIA Symp 17–21 (2001).
Dubey, A. et al. The Llama 3 Herd of Models. http://arxiv.org/abs/2407.21783 (2024).
Ezzell, G. et al. Common error pathways seen in the RO-ILS data that demonstrate opportunities for improving treatment safety. Pr. Radiat. Oncol. 8, 123–132 (2018).
Hurkmans, C. et al. A joint ESTRO and AAPM guideline for development, clinical validation and reporting of artificial intelligence models in radiation therapy. Radiother. Oncol. 197, 110345 (2024).
RLDatix. Event Reporting. https://rldatix.com/en-nam/solutions/how-we-help/risk/event-reporting/.
Alsentzer, E. et al. Publicly Available Clinical BERT Embeddings. http://arxiv.org/abs/1904.03323 (2019).
Liu, Y. et al. RoBERTa: A Robustly Optimized BERT Pretraining Approach. http://arxiv.org/abs/1907.11692 (2019).
Kailas, P., Goto, S., Homilius, M., MacRae, C. A. & Deo, R. C. Robust DeID: De-Identification of Medical Notes using Transformer Architectures. https://pypi.org/project/robust-deid/ (2022).
Fan, J.-W. & Friedman, C. Word sense disambiguation via semantic type classification. AMIA Annu Symp. Proc. 2008, 177–181 (2008).
Rios, A. M. PYMETAMAP. https://github.com/AnthonyMRios/pymetamap.
Hugging Face. PEFT (Parameter-Efficient Fine-Tuning). https://huggingface.co/docs/peft/index.
Hu, E. J. et al. LoRA: Low-rank adaptation of large language models. https://doi.org/10.48550/arXiv.2106.09685 (2021).
Nvidia Corporation. NVIDIA A100 Tensor Core GPU. https://www.nvidia.com/en-us/data-center/a100/?srsltid=AfmBOoqVawK32CuqvYk3XmDuvfXewcDMEEAWkXbjyb4hF8O3XVe459UR.
Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. http://arxiv.org/abs/1810.04805 (2019).
Li, M., Gao, Q. & Yu, T. Kappa statistic considerations in evaluating inter-rater reliability between two raters: Which, when and context matters. BMC Cancer 23, 799 (2023).
Acknowledgements
This work was funded in part by the MSKCC Support Grant P30 CA008748. We acknowledge the support of the High-Performance Computing Group in the Department of Digital Informatics & Technology Solutions at MSKCC for providing the computing infrastructure and resources necessary for this project. We also extend our gratitude to Esther Rulnick from the Division of Quality and Safety at MSKCC for their assistance in extracting incidents from the RISQ database.
Author information
Authors and Affiliations
Contributions
A.J.J., K.C., A.L., and J.M.M. conceptualized the study, directly accessed, and verified the data, conducted the formal analysis and methodology, administered the project, created the software, visualized the data, wrote the original draft of the manuscript, and reviewed and edited the manuscript. S.L., C.D.B., E.H., E.L., R.M., D.P., J.C., C.P., M.G., and J.F. contributed to data curation, methodology, and review and editing of the manuscript. All authors had full access to all data in the study and had final responsibility for the decision to submit for publication.
Corresponding authors
Ethics declarations
Competing interests
The authors declare no competing interests related to this work. EH declares the following disclosures: Volunteering position on the executive board of the Knickerbocker Chapter NSDAR. J.M.M. declares the following disclosures: Grants from Varian received during my time at the University of Michigan. Honoraria paid by the Connecticut Area Medical Physics Society. Patient issued for Combined radiation acoustics and ultrasound for radiotherapy guidance and cancer targeting. Co-founder and board member at Prexient, Inc. Chair of Work Group on Science Council EDI at the American Association of Physicists in Medicine. Chair of Work Group on Report Writing at the American Association of Physicists in Medicine. Vice Chair of Research Committee at the Radiation Oncology Institute. Co-Chair at the Radiation Oncology Institute Radiation Oncology Safety Stakeholders Initiative. FuseOncology (Copyright), which has been licensed by my previous institution (University of Michigan) for which I made a contribution. Consultant (unfunded) to the Michigan Radiation Oncology Quality Consortium.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
About this article
Cite this article
Jinia, A.J., Chapman, K., Liu, S. et al. Artificial intelligence-based incident analysis and learning system to enhance patient safety and improve treatment quality. npj Digit. Med. (2026). https://doi.org/10.1038/s41746-026-02390-2
Received:
Accepted:
Published:
DOI: https://doi.org/10.1038/s41746-026-02390-2
