Eye tracking based deep learning analysis for the early detection of diabetic retinopathy: A pilot study

Jiang，Hongyang1; Hou，Yilin1; Miao，Hanpei2; Ye，Haili2; Gao，Mengdi3; Li，Xiaoling5; Jin，Richu1; Liu，Jiang1,2,4,5,6

2023-07-01

10.1016/j.bspc.2023.104830

Biomedical Signal Processing and Control   影响因子和分区

1746-8094

1746-8108

Deep neural networks (DNNs) have exhibited impressive performance in the diabetic retinopathy (DR) computer-aided diagnosis (CAD) systems. However, DNNs are quite hungry for enormous labeled images. The limited data volume may degrade both the accuracy and interpretability of DNNs seriously. To alleviate this situation, it is significant to excavate valuable prior information. We aim to explore how to utilize ophthalmologist's eye tracking information into an early DR detection system thus to improve the classification accuracy and interpretability. In this paper, ophthalmologists’ gaze maps are firstly collected from their eye movements through eye tracker during DR diagnosis. Then we investigate an eye tracking based early DR detection model based on ophthalmologists’ gaze maps. First, we analysis the effect of the gaze map integrated with the original fundus image based on two image fusion approaches. Second, the weighted gaze map is regarded as a supervised mask to guide the learning of the attention of a DNN model. Additionally, we propose a novel difficulty-aware and class-adaptive gaze map attention learning strategy to enhance the model interpretability. Comparative experiments prove that the weighted gaze map contains more medical knowledge for diagnostic decision. Meanwhile, the attention guidance method via class activate map (CAM) regularization demonstrates its superiority on improving both the accuracy and interpretability of early DR detection model. These investigations indicate that ophthalmologists’ gaze maps, as medical prior knowledge, can contribute to the design of early DR detection model.

Class activate map Computer-aided diagnosis Diabetic retinopathy Eye tracking Gaze map

SCI ; EI

英语

第一 ; 通讯

General Program of National Natural Science Foundation of China[82272086] ; National Natural Science Foundation of China[62101236] ; Guang-dong Provincial Department of Education[2020ZDZX3043] ; Guangdong Provincial Key Laboratory[2020B121201001] ; Shenzhen Natural Science Fund[JCYJ20200109140820699] ; null[20200925174052004]

Engineering

Engineering, Biomedical

WOS:000955125500001

ELSEVIER SCI LTD

20231113709220

Classification (of information) ; Computer aided analysis ; Computer aided diagnosis ; Computer aided instruction ; Deep neural networks ; Eye movements ; Eye protection ; Image fusion ; Learning systems ; Ophthalmology

Biomedical Engineering:461.1 ; Ergonomics and Human Factors Engineering:461.4 ; Medicine and Pharmacology:461.6 ; Information Theory and Signal Processing:716.1 ; Data Processing and Image Processing:723.2 ; Computer Applications:723.5 ; Education:901.2 ; Information Sources and Analysis:903.1 ; Accidents and Accident Prevention:914.1

2-s2.0-85149730903

Scopus

1.Department of Computer Science and Engineering,Southern University of Science and Technology,Shenzhen,518055,China
2.Research Institute of Trustworthy Autonomous Systems,Southern University of Science and Technology,Shenzhen,518055,China
3.Department of Biomedical Engineering,College of Future Technology,Peking University,Beijing,100091,China
4.Singapore Eye Research Institute,Singapore National Eye Centre,Singapore,169856,Singapore
5.School of Ophthalmology and Optometry,Wenzhou Medical University,Wenzhou,325035,China
6.Guangdong Provincial Key Laboratory of Brain-inspired Intelligent Computation,Department of Computer Science and Engineering,Southern University of Science and Technology,Shenzhen,518055,China

Jiang，Hongyang,Hou，Yilin,Miao，Hanpei,et al. Eye tracking based deep learning analysis for the early detection of diabetic retinopathy: A pilot study[J]. Biomedical Signal Processing and Control,2023,84.

Jiang，Hongyang.,Hou，Yilin.,Miao，Hanpei.,Ye，Haili.,Gao，Mengdi.,...&Liu，Jiang.(2023).Eye tracking based deep learning analysis for the early detection of diabetic retinopathy: A pilot study.Biomedical Signal Processing and Control,84.

Jiang，Hongyang,et al."Eye tracking based deep learning analysis for the early detection of diabetic retinopathy: A pilot study".Biomedical Signal Processing and Control 84(2023).