Deep learning (DL) has been touted as a paradigm-shifting field of research with potential to transform the provision of eye care. With the dramatic rise of complex clinical data, DL solutions have been proposed to automate the data analysis process with an eye to decrease clinician burden. More recently, innovative applications of DL aim to assist clinicians with patient management. Jason Charng explores current and novel applications of DL in retinal diseases.
Deep learning, a term often erroneously used interchangeably with artificial intelligence or machine learning, aims to imitate the way humans process information. This is primarily done by exposing the DL algorithm to correctly annotated data and letting the program develop its own rationale to complete the task at hand. For example, if one was to develop a DL algorithm to classify birds from mammals, one would first input correctly labelled images of both animal classes into the algorithm. The DL algorithm would then identify and learn features it deems to differentiate these two groups by itself. One such feature may be that birds have beaks.
Novel applications include predicting DR development using fundus photographs and clinical risk factors,12 and estimating DR progression using fundus photographs alone
Once the algorithm is trained, new images can be fed into it to identify whether the animal is avian or mammalian.
DL has been extensively used in image analysis due to its ability to process and extract features directly from the input image. The trained algorithm can then be applied to the identification, classification and segmentation of images.1
Image segmentation, which classifies different portions of the image into groups or categories that share similar characteristics, is a time-consuming task in the clinic that can be streamlined with DL solutions. Examples of image segmentation include delineating diseased area in an en face image2 and sectioning different layers of an optical coherence tomography (OCT) B-scan.3
Diabetic Retinopathy
Early detection of diabetic retinopathy (DR) is essential in the prevention of vision loss. One effective public health strategy in early DR detection is an efficient screening program, which requires evaluating fundus images from patients. The current gold standard in telemedicine-based DR programs relies on human grading of fundus images taken from satellite sites.4-6 This time-consuming task draws clinicians away from precious chair time.
DL-based algorithms, designed to detect DR in fundus images, have been shown to have very similar accuracy compared to human graders.7,8 They have subsequently received regulatory approval for deployment into the health system (i.e. IDx-DR, United States; RetCad, European Union; SELENA+, Singapore).
Beyond disease detection, DL systems have also been developed to grade DR severity of fundus images,9 which may assist clinical decision making. Furthermore, investigators have deployed DL to analyse other multimodal images, such as optical coherence tomography (OCT)10 and OCT angiography (OCT-A),11 which can further alleviate the time-consuming demand of human grading. One exciting area of research is the potential to use DL to create personalized treatment, which should translate to better patient outcomes. Novel applications include predicting DR development using fundus photographs and clinical risk factors,12 and estimating DR progression using fundus photographs alone.13 Figure 1 summarises the current applications of DL in DR.
Figure 1. Current applications of deep learning in diabetic retinopthy.
Age-Related Macular Degeneration
Currently, the majority of DL applications in age-related macular degeneration (AMD) centre on the disease diagnosis using multimodal images. DL algorithms have been developed to determine whether a patient requires ophthalmology referral, based on assessing their fundus colour photograph, with the accuracy of these algorithms comparable to human graders.14
Additionally, algorithms have been developed to diagnose the presence of AMD using OCTs.15,16 Importantly, it has been shown that an AMD diagnosis algorithm using both fundus photographs and OCTs was more accurate than using one image modality alone,17 which is more reflective of everyday practice.
Investigators have also explored more complicated applications of DL beyond disease detection. Algorithms have been developed to classify AMD fundus images into clinical diagnoses of normal, nonexudative, or exudative AMD,18,19 which can automatically flag cases of interest to the practitioner. Further advancements in image classification include categorizing fundus images into both the nine-step20 and the simplified four-step21 Age-Related Eye Disease Study (AREDS) scales. These applications provide the ability to process large amounts of data while freeing up clinician time.
DL has also been used in more profound AMD applications. It has been employed to perform the time-consuming task of segmenting retinal layers in AMD OCT images.22 More interesting applications include identifying novel OCT-based disease biomarkers, such as pachychoroid-related changes23 and hyperreflective foci.24
An exciting field of research is using DL to predict AMD progression. Currently, mathematical models derived from population-based longitudinal data are used to predict the risk of AMD progression to ensure appropriate review periods and timely treatment.25,26 DL has been employed to both process the large amount of clinical data as well as develop the model itself.27,28 Impressively, it has been shown that a DL-based prediction model provided higher prognostic accuracy in predicting the risk of developing late-AMD than existing non-DL clinical standards.29
Furthermore, a DL-based prediction model combining both genetic and fundus data showed a better prediction of patients developing AMD than using fundus data alone.30 This opens the exciting possibility of using DL to amalgamate large amounts of diverse clinical data to better triage patients and enhance patient management.
Inherited Retinal Diseases
Similar to DR and AMD, to date, the majority of DL applications in inherited retinal diseases (IRD) have focused on image analysis. Examples include detecting the presence of pigmentary changes in colour fundus photographs of patients with retinitis pigmentosa31 and identifying patients with Stargardt disease in OCTs.32 Advancements have been made in developing DL algorithms delineating retinal layers in eyes with IRD,33 which can process large amounts of data efficiently while eliminating human bias. Furthermore, DL algorithms have been shown to analyse disease progression in IRD eyes by quantifying retinal changes over time in achromatopsia,34 retinitis pigmentosa,34,35 and Stargardt disease.2,34
Given that novel gene therapy options are in the pipeline to treat IRD, one question pertinent in choosing patients for future clinical trials is what the potential functional improvement will be following treatment Investigators have shown it is possible to predict visual acuity using confocal scanning laser ophthalmoscopy in retinitis pigmentosa eyes36 and retinal sensitivity from OCT scans in blue cone monochromacy and Leber congenital amaurosis.37,38 Hence these algorithms can assist investigators in identifying appropriate patients (i.e., those with greatest potential for vision improvement) for treatment as well as counselling patient expectations.
Rise of the Machines?
A major concern for clinicians is the notion of DL taking over human-based consultation. However, it is evident from current literature that DL algorithms are designed to streamline and/or assist clinical decision making and free up clinician time.
We note that DL algorithms face two major obstacles before being adapted into everyday practice.
Firstly, the features that a DL algorithm extracts from the data to reach its decision are mainly unknown to the human operator, hence human interpretation of the logic derived by the algorithm is complex. This differs from traditional clinical diagnosis, which is based on a logical compilation of clinical data. Therefore, clinicians are wary of clinical decision making based on DL algorithms alone.
Secondly, it has been shown that variability in algorithm designing can lead to difference in performance. Sources of variability include the classification of disease made by human graders, the differences in population demographics, the clinical data used, and the algorithm architecture. Therefore, there is a need to benchmark algorithms before clinical deployment.
Research is underway to address the two aforementioned obstacles to increase the acceptance of the technology.
In conclusion, DL is a fast-evolving area of study with widespread applications that shows promise to transform eye care. With aims to decrease clinician load and improve patient management, the technology offers revolutionary advances that will invariably be integrated into everyday practice.
Jason Charng is a senior lecturer at the Department of Optometry, University of Western Australia and Research Fellow at Lions Eye Institute. He has a keen interest in technological advances assessing structure and function of the retina, in particular as outcome measures for clinical trials.
References
1. Shen D, Wu G, Suk HI. Deep learning in medical image analysis. Annu Rev Biomed Eng 2017;19:221-248.
2. Charng J, Xiao D, Mehdizadeh M, et al., Deep learning segmentation of hyperautofluorescent fleck lesions in Stargardt disease. Sci Rep 2020;10:16491.
3. Kugelman J, Allman J, Read SA, et al., A comparison of deep learning U-Net architectures for posterior segment OCT retinal layer segmentation. Sci Rep 2022;12:14888.
4. Harper CA, Livingston PM, Wood C, et al., Screening for diabetic retinopathy using a non-mydriatic retinal camera in rural Victoria. Aust N Z J Ophthalmol 1998;26:117-121.
5. Murray RB, Metcalf SM, Lewis PM, et al., Sustaining remote-area programs: retinal camera use by Aboriginal health workers and nurses in a Kimberley partnership. Med J Aust 2005;182:520-523.
6. Nguyen HV, Tan GS, Tapp RJ, et al., Cost-effectiveness of a national telemedicine diabetic retinopathy screening program in Singapore. Ophthalmology 2016;123:2571-2580.
7. Abramoff MD, Lou Y, Erginay A, et al., Improved Automated detection of diabetic retinopathy on a publicly available dataset through integration of deep learning. Invest Ophthalmol Vis Sci 2016;57:5200-5206.
8. Gulshan V, Peng L, Coram M, et al., Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 2016;316:2402-2410.
9. Abramoff MD, Lavin PT, Birch M, et al., Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices. NPJ Digit Med 2018;1:39.
10. ElTanboly A, Ismail M, Shalaby A, et al., A computer-aided diagnostic system for detecting diabetic retinopathy in optical coherence tomography images. Med Phys 2017;44:914-923.
11. Heisler M, Karst S, Lo J, et al. Ensemble deep learning for diabetic retinopathy detection using optical coherence tomography angiography. Transl Vis Sci Technol 2020;9:20.
12. Bora A, Balasubramanian S, Babenko B, et al., Predicting the risk of developing diabetic retinopathy using deep learning. Lancet Digit Health 2021;3:e10-e19.
13. Arcadu F, Benmansour F, Maunz A, et al., Deep learning algorithm predicts diabetic retinopathy progression in individual patients. NPJ Digit Med 2019;2:92.
14. Li F, Liu Z, Chen H, Jiang M, Zhang X, Wu Z. Automatic Detection of Diabetic Retinopathy in Retinal Fundus Photographs Based on Deep Learning Algorithm. Transl Vis Sci Technol 2019;8:4.
15. Lee CS, Baughman DM, Lee AY. Deep learning is effective for the classification of OCT images of normal versus age-related macular degeneration. Ophthalmol Retina 2017;1:322-327.
16. Treder M, Lauermann JL, Eter N. Automated detection of exudative age-related macular degeneration in spectral domain optical coherence tomography using deep learning. Graefes Arch Clin Exp Ophthalmol 2018;256:259-265.
17. Yoo TK, Choi JY, Seo JG, et al., The possibility of the combination of OCT and fundus images for improving the diagnostic accuracy of deep learning for age-related macular degeneration: a preliminary experiment. Med Biol Eng Comput 2019;57:677-687.
18. An G, Akiba M, Yokota H, et al., Deep learning classification models built with two-step transfer learning for age related macular degeneration diagnosis. Annu Int Conf IEEE Eng Med Biol Soc 2019;2019:2049-2052.
19. Motozawa N, An G, Takagi S, et al., Optical coherence tomography-based deep-learning models for classifying normal and age-related macular degeneration and exudative and non-exudative age-related macular degeneration changes. Ophthalmol Ther 2019;8:527-539.
20. Grassmann F, Mengelkamp J, Brandl C, et al., A deep learning algorithm for prediction of age-related eye disease study severity scale for age-related macular degeneration from color fundus photography. Ophthalmology 2018;125:1410-1420.
21. Peng Y, Dharssi S, Chen Q, et al., DeepSeeNet: A deep learning model for automated classification of patient-based age-related macular degeneration severity from color fundus photographs. Ophthalmology 2019;126:565-575.
22. Schmidt-Erfurth U, Vogl WD, Jampol LM, Bogunovic H. Application of automated quantification of fluid volumes to anti-vegf therapy of neovascular age-related macular degeneration. Ophthalmology 2020;127:1211-1219.
23. Hosoda Y, Miyake M, Yamashiro K, et al., Deep phenotype unsupervised machine learning revealed the significance of pachychoroid features in etiology and visual prognosis of age-related macular degeneration. Sci Rep 2020;10:18423.
24. Saha S, Nassisi M, Wang M, et al. Automated detection and classification of early AMD biomarkers using deep learning. Sci Rep 2019;9:10990.
25. Chiu CJ, Mitchell P, Klein R, et al., A risk score for the prediction of advanced age-related macular degeneration: development and validation in two prospective cohorts. Ophthalmology 2014;121:1421-1427.
26. Joachim ND, Mitchell P, Kifley A, Wang JJ. Incidence, progression, and associated risk factors of medium drusen in age-related macular degeneration: findings from the 15-year follow-up of an Australian cohort. JAMA Ophthalmol 2015;133:698-705.
27. Bhuiyan A, Wong TY, Ting DSW, et al., Artificial intelligence to stratify severity of age-related macular degeneration (AMD) and predict risk of progression to late AMD. Transl Vis Sci Technol 2020;9:25.
28. Burlina PM, Joshi N, Pacheco KD, Freund DE, Kong J, Bressler NM. Use of Deep Learning for Detailed Severity Characterization and Estimation of 5-Year Risk Among Patients With Age-Related Macular Degeneration. JAMA Ophthalmol 2018;136:1359-1366.
29. Peng Y, Keenan TD, Chen Q, et al. Predicting risk of late age-related macular degeneration using deep learning. NPJ Digit Med 2020;3:111.
30. Yan Q, Weeks DE, Xin H, et al., Deep-learning-based prediction of late age-related macular degeneration progression. Nat Mach Intell 2020;2:141-150.
31. Arsalan M, Baek NR, Owais M, et al., Deep learning-based detection of pigment signs for analysis and diagnosis of retinitis pigmentosa. Sensors (Basel) 2020;20.
32. Shah M, Roomans Ledo A, Rittscher J. Automated classification of normal and Stargardt disease optical coherence tomography images using deep learning. Acta Ophthalmol 2020;98:e715-e721.
33. Kugelman J, Alonso-Caneiro D, Chen Y, et al., Retinal boundary segmentation in Stargardt disease optical coherence tomography images using automated deep learning. Transl Vis Sci Technol 2020;9:12.
34. Davidson B, Kalitzeos A, Carroll J, et al., Automatic cone photoreceptor localisation in healthy and Stargardt afflicted retinas using deep learning. Sci Rep 2018;8:7911.
35. Wang YZ, Wu W, Birch DG. A hybrid model composed of two convolutional neural networks for automatic retinal layer segmentation of OCT images in retinitis pigmentosa. Transl Vis Sci Technol 2021;10:9.
36. Liu TYA, Ling C, Hahn L, et al., Prediction of visual impairment in retinitis pigmentosa using deep learning and multimodal fundus images. Br J Ophthalmol 2022.
37. Sumaroka A, Cideciyan AV, Sheplock R, et al., Foveal therapy in blue cone monochromacy: predictions of visual potential from artificial intelligence. Front Neurosci 2020;14:800.
38. Sumaroka A, Garafalo AV, Semenov EP, et al., Treatment potential for macular cone vision in Leber Congenital Amaurosis due to CEP290 or NPHP5 mutations: predictions from artificial intelligence. Invest Ophthalmol Vis Sci 2019;60:2551-2562.
