Real-world validation of artificial intelligence algorithms for ophthalmic imaging

With the explosion of artificial intelligence (AI) algorithms in medical imaging, lifecycle AI development is well accepted and includes training, internal validation, external validation. Although training validation can be done retrospective datasets, that testing using independent data imperative. These requirements overcome basic issue model overfitting, wherein algorithm tends to perform within environment, but accuracy not sustained stage. Validation for imaging studies need representative target population consider various factors, such as geographical temporal disease prevalence, racial gender diversity, camera systems, image specifications, acquisition specifications. Despite datasets a deep-learning diabetic-retinopathy screening developed by Google Health faced challenges when deployed clinic workflow.1Gulshan V Peng L Coram M et al.Development deep learning detection diabetic retinopathy retinal fundus photographs.JAMA. 2016; 316: 2402-2410Crossref PubMed Scopus (2685) Scholar, 2Beede E, Baylor Hersch F, al. A human-centered evaluation system clinics retinopathy. Proceedings 2020 CHI Conference on Human Factors Computing Systems; Honolulu, HI, USA; 2020.Google Scholar After some have been tested real-world dataset dataset, they found bias performance with variable depending pigmentation.3Burlina P Joshi N Paul W al.Addressing diagnostics.Transl Vis Sci Technol. 2021; 10: 13Crossref (4) The US Food Drug Administration (FDA) requires use prospective final testing, was case two FDA-approved models retinopathy.4Abràmoff MD Lavin PT Birch al.Pivotal trial an autonomous AI-based diagnostic primary care offices.NPJ Digit Med. 2018; 1: 39Crossref 5EYENUKEyenuk announces FDA clearance EyeArt screening.https://www.eyenuk.com/us-en/articles/diabetic-retinopathy/eyenuk-announces-eyeart-fda-clearance/Date accessed: June 24, 2021Google There dearth clinical rigorously validating real world, more designed studies, ideally locked algorithm, are needed truly understand strengths drawbacks given practice.6Nagendran Chen Y Lovejoy CA al.Artificial versus clinicians: systematic review design, reporting standards, claims studies.BMJ. 2020; 368: m689Crossref (196) Several state-of-the-art automated systems were compared head world showed differences sensitivities ranging between 50·98% 85·90%.7Lee AY Yanagihara RT Lee CS al.Multicenter, head-to-head, study seven systems.Diabetes Care. 441168Crossref (15) This emphasised multiple levels diversity careful consideration regarding reference standards which output being compared. Article Duoru Lin coworkers8Lin D Xiong J Liu C al.Application Comprehensive Artificial Retinal Expert (CARE) system: national evidence study.Lancet Health. 3: e486-e495Summary Full Text PDF (3) Lancet Digital features called system, single convolutional neural network trained identify 14 common abnormalities normal fundus. methodologically planned include labelling ophthalmologists experts varying yearsof experience. All standardised grading triple read arbitration method. More than 200 000 images obtained from 16 settings across China, including tertiary hospitals, community physical examination centers. comprehensively validated 18 136 prospectively collected China 35 sites, again covering range settings. Performance CARE nine experience standard four groups experiences. In addition, participants non-Chinese ethnicities different cameras. Mean 0·968 (0·037), similar ethnicities. multidisease also single-disease-labelled binary better (mean AUC 0·952, SD 0·047 vs 0·921, 0·087). authors explained all labels into one enables learn correlation logic, conditions drusen, neovascularisation, geographic atrophy. They concluded satisfactory could implemented care. diverse appears best possible way test models, implementation workflow separate important factor evaluate.9He Baxter SL Xu al.The practical technologies medicine.Nat 2019; 25: 30-36Crossref (385) coworkers,8Lin do discuss purpose triaging specialty clinics. there other factors workflow, spectrum affect will considered step. most development, focus metrics establish or abnormality classification. Quality assessment integral classification based at stages development. this study, quality-control applied during introduced so-called uncertain category needs implemented, particularly classifiers evolve several diseases.10Kompa B Snoek Beam AL Second opinion needed: communicating uncertainty machine learning.NPJ 4: 4Crossref (25) Model does necessarily translate improvement patient outcomes. algorithms, burden placed teleophthalmology health-care staff relieved. However, outcomes reduction retinopathy-related vision loss yet established. Development recognises imply reaches ophthalmologist needed, work it step right direction. We declare no competing interests. Application studyOur DLS photographs, so allow adopted Full-Text Open Access