Abstract
Objectives
To evaluate the performance of an artificial intelligence (AI) system (Pegasus, Visulytix Ltd., UK*) at the detection of diabetic retinopathy (DR) from images captured by a handheld portable fundus camera.
Methods
A cohort of 6404 patients (~80% with diabetes mellitus) was screened for retinal diseases using a handheld portable fundus camera (Pictor Plus, Volk Optical Inc., USA) at the Mexican Advanced Imaging Laboratory for Ocular Research. The images were graded for DR by specialists according to the Scottish DR grading scheme. The performance of the AI system was evaluated, retrospectively, in assessing referable DR (RDR) and proliferative DR (PDR) and compared with the performance on a publicly available desktop camera benchmark dataset.
Results
For RDR detection, Pegasus performed with an 89.4% (95% CI: 88.0–90.7) area under the receiver operating characteristic (AUROC) curve for the MAILOR cohort, compared with an AUROC of 98.5% (95% CI: 97.8–99.2) on the benchmark dataset. This difference was statistically significant. Moreover, no statistically significant difference was found in performance for PDR detection with Pegasus achieving an AUROC of 94.3% (95% CI: 91.0–96.9) on the MAILOR cohort and 92.2% (95% CI: 89.4–94.8) on the benchmark dataset.
Conclusions
Pegasus showed good transferability for the detection of PDR from a curated desktop fundus camera dataset to real-world clinical practice with a handheld portable fundus camera. However, there was a substantial, and statistically significant, decrease in the diagnostic performance for RDR when using the handheld device.
Similar content being viewed by others
Log in or create a free account to read this content
Gain free access to this article, as well as selected content from this journal and more on nature.com
or
Data availability
The public benchmark dataset (IDRiD) is available on a CC-BY 4.0 license. The data collected in the MAILOR dataset are available at the discretion of MAILOR.
References
Bergman M, Buysschaert M, Schwarz PE, Albright A, Narayan KV, Yach D. Diabetes prevention: global health policy and perspectives from the ground. Diabetes Manag. 2012;2:309.
Royal College of Ophthalmologists. Workforce Census 2016 [report]. 2016. https://www.rcophth.ac.uk/wp-content/uploads/2017/03/RCOphth-Workforce-Census-2016.pdf.
Resnikoff S, Felch W, Gauthier TM, Spivey B. The number of ophthalmologists in practice and training worldwide: a growing gap despite more than 200 000 practitioners. Br J Ophthalmol. 2012;96:783–7.
Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017;542:115.
Becker AS, Marcon M, Ghafoor S, Wurnig MC, Frauenfelder T, Boss A. Deep learning in mammography: diagnostic accuracy of a multipurpose image analysis software in the detection of breast cancer. Investigative Radiol. 2017;52:434–40.
Abràmoff MD, Garvin MK, Sonka M. Retinal imaging and image analysis. IEEE Rev Biomed Eng. 2010;3:169–208.
Porwal P, Pachade S, Kamble R, Kokare M, Deshmukh G, Sahasrabuddhe V, et al. Indian diabetic retinopathy image dataset (IDRiD): a database for diabetic retinopathy screening research. Data. 2018;3:25.
Gosheva M, Klameth C, Norrenberg L, Clin L, Dietter J, Haq W, et al. Quality and learning curve of handheld versus stand-alone non-mydriatic cameras. Clin Ophthalmol. 2017;11:1601.
Zachariah S, Wykes W, Yorston D. Grading diabetic retinopathy (DR) using the Scottish grading protocol. Community Eye Health. 2015;28:72.
American Academy of Ophthalmology. International Clinical Diabetic Retinopathy Disease Severity Scale. 2002 http://www.icoph.org/downloads/Diabetic-Retinopathy-Scale.pdf
Rajalakshmi R, Subashini R, Anjana RM, Mohan V. Automated diabetic retinopathy detection in smartphone-based fundus photography using artificial intelligence. Eye. 2018;32:1138.
Tufail A, Rudisill C, Egan C, Kapetanakis VV, Salas-Vega S, Owen CG. et al. Automated diabetic retinopathy image assessment software: diagnostic accuracy and cost-effectiveness compared with human graders. Ophthalmology. 2017;124:343–51.
Ting DS, Cheung CY, Lim G, Tan GS, Quang ND, Gan A, et al. Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. J Am Med Assoc. 2017;318:2211–23.
Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. J Am Med Assoc. 2016;316:2402–10.
Acknowledgements
The authors acknowledge Visulytix Ltd.* for provision of the deep learning system, Pegasus, and thank Dr Kanwal Bhatia for her contribution.
Funding
Funding was provided by Visulytix Ltd.* in the provision of the Pegasus deep learning software and statistical analysis of results. *Visulytix Ltd. is currently in liquidation.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
TWR, JG-B and NJ were employed by, and owned stock options in Visulytix Ltd.* ST owned shares in Visulytix Ltd and received honoraria from Visulytix Ltd.* JV, VCL, ELS, DMM and RGF declare no conflicts of interest.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Rights and permissions
About this article
Cite this article
Rogers, T.W., Gonzalez-Bueno, J., Garcia Franco, R. et al. Evaluation of an AI system for the detection of diabetic retinopathy from images captured with a handheld portable fundus camera: the MAILOR AI study. Eye 35, 632–638 (2021). https://doi.org/10.1038/s41433-020-0927-8
Received:
Revised:
Accepted:
Published:
Issue date:
DOI: https://doi.org/10.1038/s41433-020-0927-8
This article is cited by
-
Performance and limitation of machine learning algorithms for diabetic retinopathy screening and its application in health management: a meta-analysis
BioMedical Engineering OnLine (2025)
-
The efficacy of artificial intelligence in diabetic retinopathy screening: a systematic review and meta-analysis
International Journal of Retina and Vitreous (2025)
-
Diabetic retinopathy: a comprehensive review of pathophysiology and emerging treatments
Molecular Biology Reports (2025)
-
Artificial intelligence for early detection of diabetes mellitus complications via retinal imaging
Journal of Diabetes & Metabolic Disorders (2025)
-
Artificial Intelligence for Chronic Disease Screening in Latin America and the Caribbean: The Diagnostic Potential of Digital Health
Journal of Medical Systems (2025)
