Abstract
Purpose
The aim of this study is to investigate the efficacy of a mobile platform that combines smartphone-based retinal imaging with automated grading for determining the presence of referral-warranted diabetic retinopathy (RWDR).
Methods
A smartphone-based camera (RetinaScope) was used by non-ophthalmic personnel to image the retina of patients with diabetes. Images were analyzed with the Eyenuk EyeArt® system, which generated referral recommendations based on presence of diabetic retinopathy (DR) and/or markers for clinically significant macular oedema. Images were independently evaluated by two masked readers and categorized as refer/no refer. The accuracies of the graders and automated interpretation were determined by comparing results to gold standard clinical diagnoses.
Results
A total of 119 eyes from 69 patients were included. RWDR was present in 88 eyes (73.9%) and in 54 patients (78.3%). At the patient-level, automated interpretation had a sensitivity of 87.0% and specificity of 78.6%; grader 1 had a sensitivity of 96.3% and specificity of 42.9%; grader 2 had a sensitivity of 92.5% and specificity of 50.0%. At the eye-level, automated interpretation had a sensitivity of 77.8% and specificity of 71.5%; grader 1 had a sensitivity of 94.0% and specificity of 52.2%; grader 2 had a sensitivity of 89.5% and specificity of 66.9%.
Discussion
Retinal photography with RetinaScope combined with automated interpretation by EyeArt achieved a lower sensitivity but higher specificity than trained expert graders. Feasibility testing was performed using non-ophthalmic personnel in a retina clinic with high disease burden. Additional studies are needed to assess efficacy of screening diabetic patients from general population.
Similar content being viewed by others
Log in or create a free account to read this content
Gain free access to this article, as well as selected content from this journal and more on nature.com
or
References
IDF diabetes atlas - 2017 Atlas. 2018. http://diabetesatlas.org/resources/2017-atlas.html.
UK Prospective Diabetes Study (UKPDS) Group. Intensive blood-glucose control with sulphonylureas or insulin compared with conventional treatment and risk of complications in patients with type 2 diabetes (UKPDS 33). Lancet Lond Engl. 1998;352:837–53.
Nathan DM, Genuth S, Lachin J, Cleary P, Crofford O, Davis M. Diabetes Control and Complications Trial Research Group et al. The effect of intensive treatment of diabetes on the development and progression of long-term complications in insulin-dependent diabetes mellitus. N. Engl J Med. 1993;329:977–86.
Lachin JM, White NH, Hainsworth DP, Sun W, Cleary PA, Nathan DM. Diabetes Control and Complications Trial (DCCT)/Epidemiology of Diabetes Interventions and Complications (EDIC) Research Group Effect of intensive diabetes therapy on the progression of diabetic retinopathy in patients with type 1 diabetes: 18 years of follow-up in the DCCT/EDIC. Diabetes. 2015;64:631–42.
Antonetti DA, Klein R, Gardner TW. Diabetic retinopathy. N. Engl J Med. 2012;366:1227–39. Mar 29.
Ajlan RS, Silva PS, Sun JK. Vascular endothelial growth factor and diabetic retinal disease. Semin Ophthalmol. 2016;31:40–8.
Osaadon P, Fagan XJ, Lifshitz T, Levy J. A review of anti-VEGF agents for proliferative diabetic retinopathy. Eye. 2014;28:510–20
Gündüz K, Bakri SJ. Management of proliferative diabetic retinopathy. Compr Ophthalmol Update. 2007;8:245–56.
Boyer DS, Yoon YH, Belfort R, Bandello F, Maturi RK, Augustin AJ, et al. Three-year, randomized, sham-controlled trial of dexamethasone intravitreal implant in patients with diabetic macular edema. Ophthalmology. 2014;121:1904–14.
Early Treatment Diabetic Retinopathy Study Research Group Early photocoagulation for diabetic retinopathy. ETDRS report number 9.Ophthalmology . 1991;98:766–85.
Early Treatment Diabetic Retinopathy Study research group. Photocoagulation for diabetic macular edema. Early Treatment Diabetic Retinopathy Study report number 1. Arch Ophthalmol Chic Ill 1960. 1985;103:1796–806.
The Diabetic Retinopathy Study Research Group. Photocoagulation treatment of proliferative diabetic retinopathy. Clinical application of Diabetic Retinopathy Study (DRS) findings, DRS Report Number 8. Ophthalmology. 1981;88:583–600.
Ferris FL. How effective are treatments for diabetic retinopathy? JAMA. 1993;269:1290–1.
Murchison AP, Hark L, Pizzi LT, Dai Y, Mayro EL, Storey PP, et al. Non-adherence to eye care in people with diabetes. BMJ Open Diabetes Res Care. 2017;5. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5574424/.
Tao X, Li J, Zhu X, Zhao B, Sun J, Ji L, et al. Association between socioeconomic status and metabolic control and diabetes complications: a cross-sectional nationwide study in Chinese adults with type 2 diabetes mellitus. Cardiovasc Diabetol. 2016;15:61.
Wang SY, Andrews CA, Herman WH, Gardner TW, Stein JD. Incidence and risk factors for developing diabetic retinopathy among youths with type 1 or type 2 diabetes throughout the United States. Ophthalmology. 2017;124:424–30.
Association AD. Standards of medical care for patients with diabetes mellitus. Diabetes Care 1989;12:365–8.
Brechner RJ, Cowie CC, Howie LJ, Herman WH, Will JC, Harris MI. Ophthalmic examination among adults with diagnosed diabetes mellitus. JAMA 1993;270:1714–8.
Fisher MD, Rajput Y, Gu T, Singer JR, Marshall AR, Ryu S, et al. Evaluating adherence to dilated eye examination recommendations among patients with diabetes, combined with patient and provider perspectives. Am Health Drug Benefits. 2016;9:385–93.
Zhang X, Beckles GL, Chou C-F, Saaddine JB, Wilson MR, Lee PP, et al. Socioeconomic disparity in use of eye care services among US adults with age-related eye diseases: National Health Interview Survey, 2002 and 2008. JAMA Ophthalmol. 2013;131:1198–206.
Fathy C, Patel S, Sternberg P, Kohanim S. Disparities in adherence to screening guidelines for diabetic retinopathy in the United States: a comprehensive review and guide for future directions. Semin Ophthalmol. 2016;31:364–77.
Paz SH, Varma R, Klein R, Wu J, Azen SP. Los Angeles Latino Eye Study Group Noncompliance with vision care guidelines in Latinos with type 2 diabetes mellitus: the Los Angeles Latino Eye Study. Ophthalmology. 2006;113:1372–7.
Ellish NJ, Royak-Schaler R, Passmore SR, Higginbotham EJ. Knowledge, attitudes, and beliefs about dilated eye examinations among African-Americans. Invest Ophthalmol Vis Sci. 2007;48:1989–94.
Gibson DM. Eye care availability and access among individuals with diabetes, diabetic retinopathy, or age-related macular degeneration. JAMA Ophthalmol. 2014;132:471–7.
Brown M, Kuhlman D, Larson L, Sloan K, Ablah E, Konda K, et al. Does availability of expanded point-of-care services improve outcomes for rural diabetic patients? Prim Care. Diabetes. 2013;7:129–34.
Rosenberg JB, Friedman IB, Gurland JE. Compliance with screening guidelines for diabetic retinopathy in a large academic children’s hospital in the Bronx. J Diabetes Complications. 2011;25:222–6.
DeBuc DC. The role of retinal imaging and portable screening devices in tele-ophthalmology applications for diabetic retinopathy management. Curr Diab Rep. 2016;16:132.
Mansberger SL, Sheppler C, Barker G, Gardiner SK, Demirel S, Wooten K, et al. Long-term comparative effectiveness of telemedicine in providing diabetic retinopathy screening examinations: a randomized clinical trial. JAMA Ophthalmol. 2015;133:518–25.
Jani PD, Forbes L, Choudhury A, Preisser JS, Viera AJ, Garg S. Evaluation of diabetic retinal screening and factors for ophthalmology referral in a telemedicine network. JAMA Ophthalmol. 2017;135:706–14.
Kirkizlar E, Serban N, Sisson JA, Swann JL, Barnes CS, Williams MD. Evaluation of telemedicine for screening of diabetic retinopathy in the Veterans Health Administration. Ophthalmology. 2013;120:2604–10.
Zimmer-Galler IE, Kimura AE, Gupta S. Diabetic retinopathy screening and the use of telemedicine. Curr Opin Ophthalmol. 2015;26:167–72.
Liew G, Michaelides M, Bunce C. A comparison of the causes of blindness certifications in England and Wales in working age adults (16–64 years), 1999–2000 with 2009–2010. BMJ Open. 2014;4:e004015.
Toy BC, Myung DJ, He L, Pan CK, Chang RT, Polkinhorne A, et al. Smartphone-based dilated fundus photography and near visual acuity testing as inexpensive screening tools to detect referral warranted diabetic eye disease. Retin Philos Pa. 2016;36:1000–8.
Russo A, Mapham W, Turano R, Costagliola C, Morescalchi F, Scaroni N, et al. Comparison of smartphone ophthalmoscopy with slit-lamp biomicroscopy for grading vertical cup-to-disc ratio. J Glaucoma. 2016;25:e777–81.
Muiesan ML, Salvetti M, Paini A, Riviera M, Pintossi C, Bertacchini F, et al. Ocular fundus photography with a smartphone device in acute hypertension. J Hypertens. 2017;35:1660–5.
Toslak D, Ayata A, Liu C, Erol MK, Yao X. Wide-field smartphone fundus video camera based on miniaturized indirect ophthalmoscopy. Retin Philos Pa. 2018;38:438–41.
Giardini ME, Livingstone IAT, Jordan S, Bolster NM, Peto T, Burton M, et al. A smartphone based ophthalmoscope. Conf Proc Annu Int Conf IEEE Eng Med Biol Soc IEEE Eng Med Biol Soc Annu Conf. 2014;2014:2177–80.
3D printed smartphone indirect lens adapter for rapid, high quality retinal imaging. J Mobile Technol Med. 2018. https://www.journalmtm.com/2014/3d-printed-smartphone-indirect-lens-adapter-for-rapid-high-quality-retinal-imaging.
Kermany DS, Goldbaum M, Cai W, Valentim CCS, Liang H, Baxter SL, et al. Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell. 2018;172:1122–31.e9.
Poostchi M, Silamut K, Maude RJ, Jaeger S, Thoma G. Image analysis and machine learning for detecting malaria. Transl Res J Lab Clin Med. 2018;194:36–55.
Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA. 2016;316:2402–10.
Raju M, Pagidimarri V, Barreto R, Kadam A, Kasivajjala V, Aswath A. Development of a deep learning algorithm for automatic diagnosis of diabetic retinopathy. Stud Health Technol Inform. 2017;245:559–63.
Walton OB, Garoon RB, Weng CY, Gross J, Young AK, Camero KA, et al. Evaluation of Automated Teleretinal Screening Program for Diabetic Retinopathy. JAMA Ophthalmol. 2016;134:204–9.
Li Z, He Y, Keel S, Meng W, Chang RT, He M. Efficacy of a deep learning system for detecting glaucomatous optic neuropathy based on color fundus photographs. Ophthalmology. 2018;125:1199–206.
Abràmoff MD, Folk JC, Han DP, Walker JD, Williams DF, Russell SR, et al. Automated analysis of retinal images for detection of referable diabetic retinopathy. JAMA Ophthalmol. 2013;131:351–7.
Patel TP, Aaberg MT, Paulus YM, Lieu P, Dedania VS, Qian CX, et al. Smartphone-based fundus photography for screening of plus-disease retinopathy of prematurity. Graefes Arch Clin Exp Ophthalmol Albrecht Von Graefes Arch Klin Exp Ophthalmol. 2019;257:2579–85.
Patel TP, Kim TN, Yu G, Dedania VS, Lieu P, Qian CX, et al. Smartphone-based, rapid, wide-field fundus photography for diagnosis of pediatric retinal diseases. Transl Vis Sci Technol. 2019;8:29.
Ramachandran N, Hong SC, Sime MJ, Wilson GA. Diabetic retinopathy screening using deep neural network. Clin Exp Ophthalmol. 2018;46:412–6.
Xu K, Feng D, Mi H Deep. Convolutional neural network-based early automated detection of diabetic retinopathy using fundus image. Mol Basel Switz. 2017;22:2054.
Bhaskaranand M, Ramachandra C, Bhat S, Cuadros J, Nittala MG, Sadda S, et al. Automated diabetic retinopathy screening and monitoring using retinal fundus image analysis. J Diabetes Sci Technol. 2016;10:254–61.
Ting DSW, Cheung CY-L, Lim G, Tan GSW, Quang ND, Gan A, et al. Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. JAMA. 2017;318:2211–23.
Abràmoff MD, Lou Y, Erginay A, Clarida W, Amelon R, Folk JC, et al. Improved automated detection of diabetic retinopathy on a publicly available dataset through integration of deep learning. Invest Ophthalmol Vis Sci. 2016;57:5200–6.
Krause J, Gulshan V, Rahimy E, Karth P, Widner K, Corrado GS, et al. Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy. Ophthalmology. 2018;125:1264–72.
Gargeya R, Leng T. Automated identification of diabetic retinopathy using deep learning. Ophthalmology. 2017;124:962–9.
Beagley J, Guariguata L, Weil C, Motala AA. Global estimates of undiagnosed diabetes in adults. Diabetes Res Clin Pract. 2014;103:150–60.
Maamari RN, Keenan JD, Fletcher DA, Margolis TP. A mobile phone-based retinal camera for portable wide field imaging. Br J Ophthalmol. 2014;98:438–41.
Kim TN, Myers F, Reber C, Loury PJ, Loumou P, Webster D, et al. A smartphone-based tool for rapid, portable, and automated wide-field retinal imaging. Transl Vis Sci Technol. 2018;7:21–21.
Early Treatment Diabetic Retinopathy Study Research Group Grading diabetic retinopathy from stereoscopic color fundus photographs--an extension of the modified Airlie House classification. ETDRS report number 10. Ophthalmology. 1991;98:786–806.
Li P, Paulus Y, Davila J, Gosbee J, Margolis T, Fletcher D, et al. Usability testing of a smartphone-based retinal camera among first-time users in the primary care setting. BMJ Innov. 2019;5:120–6.
British Diabetic Association. Retinal photographic screening for diabetic eye disease. A British Diabetic Association Report. London: British Diabetic Association; 1997.
Scanlon PH. The English National Screening Programme for diabetic retinopathy 2003-2016. Acta Diabetol. 2017;54:515–25.
Rajalakshmi R, Subashini R, Anjana RM, Mohan V. Automated diabetic retinopathy detection in smartphone-based fundus photography using artificial intelligence. Eye Lond Engl. 2018;32:1138–44.
Russo A, Morescalchi F, Costagliola C, Delcassi L, Semeraro F. Comparison of smartphone ophthalmoscopy with slit-lamp biomicroscopy for grading diabetic retinopathy. Am J Ophthalmol. 2015;159:360–364.e1.
Ryan ME, Rajalakshmi R, Prathiba V, Anjana RM, Ranjani H, Narayan KMV, et al. Comparison Among Methods of Retinopathy Assessment (CAMRA) study: smartphone, nonmydriatic, and mydriatic photography. Ophthalmology. 2015;122:2038–43.
Vujosevic S, Benetti E, Massignan F, Pilotto E, Varano M, Cavarzeran F, et al. Screening for diabetic retinopathy: 1 and 3 nonmydriatic 45-degree digital fundus photographs vs 7 standard early treatment diabetic retinopathy study fields. Am J Ophthalmol. 2009;148:111–8.
Silva PS, Cavallerano JD, Haddad NMN, Kwak H, Dyer KH, Omar AF, et al. Peripheral lesions identified on ultrawide field imaging predict increased risk of diabetic retinopathy progression over 4 years. Ophthalmology. 2015;122:949–56.
Sun JK, Aiello LP. The future of ultrawide field imaging for diabetic retinopathy: pondering the retinal periphery. JAMA Ophthalmol. 2016;134:247–8.
Sim DA, Keane PA, Rajendram R, Karampelas M, Selvam S, Powner MB, et al. Patterns of peripheral retinal and central macula ischemia in diabetic retinopathy as evaluated by ultra-widefield fluorescein angiography. Am J Ophthalmol. 2014;158:144–153.e1.
Aptel F, Denis P, Rouberol F, Thivolet C. Screening of diabetic retinopathy: effect of field number and mydriasis on sensitivity and specificity of digital fundus photography. Diabetes Metab. 2008;34:290–3.
Murgatroyd H, Ellingford A, Cox A, Binnie M, Ellis JD, MacEwen CJ, et al. Effect of mydriasis and different field strategies on digital image screening of diabetic eye disease. Br J Ophthalmol. 2004;88:920–4.
Acknowledgements
We would like to thank the Knights Templar Eye Foundation, Research to Prevent Blindness, the Rogers Family Foundation, and the University of Michigan for their support. Additionally, we would like to thank the Eyenuk team for their willingness to partner with us and their continued commitment to ocular research.
Funding
This work was supported by the Knights Templar Eye Foundation Career-Starter Research Grant, the University of Michigan Translational Research and Commercialization for Life Sciences, the University of Michigan Center for Entrepreneurship Dean’s Engineering Translational Prototype Research Fund, the QB3 Bridging the Gap Award from the Rogers Family Foundation, the Bakar Fellows Award, the Chan-Zuckerberg Biohub Investigator Award, the National Eye Institute grant 1K08EY027458, and the University of Michigan Department of Ophthalmology and Visual Sciences.
Author information
Authors and Affiliations
Corresponding authors
Ethics declarations
Conflict of interest
DAF is a co-founder of CellScope, Inc., a company commercializing a cellphone-based otoscope, and holds shares in CellScope, Inc. DAF, TPM, CR, FM, and TNK are all inventors on the US patents and related applications pertaining to a “Retinal CellScope Apparatus”. Furthermore, MB, SB, CR, and KS are employees of Eyenuk, Inc. and are listed as inventors on the US patents and related applications pertaining to disease screening and monitoring using retinal images.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Rights and permissions
About this article
Cite this article
Kim, T.N., Aaberg, M.T., Li, P. et al. Comparison of automated and expert human grading of diabetic retinopathy using smartphone-based retinal photography. Eye 35, 334–342 (2021). https://doi.org/10.1038/s41433-020-0849-5
Received:
Revised:
Accepted:
Published:
Version of record:
Issue date:
DOI: https://doi.org/10.1038/s41433-020-0849-5
This article is cited by
-
Systematic review and meta-analysis of regulator-approved deep learning systems for fundus diabetic retinopathy detections
npj Digital Medicine (2025)
-
Automated detection of diabetic retinopathy using an improved deep learning model with smartphone images
International Journal of Diabetes in Developing Countries (2025)
-
Single retinal image for diabetic retinopathy screening: performance of a handheld device with embedded artificial intelligence
International Journal of Retina and Vitreous (2023)
-
Development and validation of a risk prediction model for diabetic retinopathy in type 2 diabetic patients
Scientific Reports (2023)
-
Deep learning for automated detection of neovascular leakage on ultra-widefield fluorescein angiography in diabetic retinopathy
Scientific Reports (2023)
