眼底图像的可泛化增强算法研究

眼底图像是获取患者视网膜信息的一种重要方法，通过分析眼底图像可以实现对大部分眼底疾病和部分全身性疾病的筛查。在临床检查中，眼底图像的成像质量对医生临床诊断和计算机自动诊断系统都十分关键，但由于眼底图像中经常会因人为操作、环境变化和病理性因素而引起图像质量下降，从而影响临床诊断的可信度。尽管目前常用的深度学习图像增强算法可以提升眼底图像质量，但是由于低质量眼底图像的采集困难、噪声的不确定性和眼底结构的复杂性等因素，导致现有的算法在临床眼底图像上表现不佳。因此，为了提升图像增强算法此类低质量眼底图像上的泛化性能，本文基于领域自适应和领域泛化分别设计了有针对性的可泛化眼底图像增强算法，研究内容主要分为：

（1）提出了基于领域自适应的白内障眼底图像增强算法。白内障患者的晶状体浑浊会引起光线散射，进而导致低质量的眼底图像，因此本文基于该原理设计成像模型，用于模拟白内障图像并提供配对的训练数据；之后通过提取高频成分取代了基于分割标签作为结构保持的模块；最后结合无监督的领域自适应，开发了适用于临床场景的白内障眼底图像增强算法。

（2）提出了基于领域泛化的白内障眼底图像增强算法。该算法首先根据白内障成像模型进行领域随机化以生成多个源域；之后通过提取高频成分去除领域特有特征，并保留领域不变特征；最后基于领域对齐网络，获取保留视网膜结构并且清晰的眼底图像，实现无需目标域的可泛化图像增强。

（3）提出了基于金字塔一致性的可泛化眼底图像增强算法。该算法首先分析了低质量眼底图像的成像模型，然后基于成像模型对清晰图像进行多次随机退化，以获得具有内容一致性的图像序列；之后利用拉普拉斯金字塔将图像分解为线性可分的多频域图像特征；最后基于退化图像序列引入的内容一致性，以及在特征金字塔上进行的特征一致性约束，实现了退化不变的可泛化图像增强。

上述三种算法层层递进，从自适应临床数据的白内障眼底图像增强，到无需目标域数据实现白内障眼底图像领域泛化的图像增强，再到基于金字塔一致性的可泛化眼底图像增强。所提出算法被用于真实临床数据的实验中，优于其它先进方法的实验结果说明了算法的泛化性以及算法在下游临床任务的有效性。

Retinal fundus images have been widely adopted in clinical fundus examination as an economical and efficient fundus imaging modality. By analyzing fundus images, most fundus and some systemic diseases can be diagnosed. In clinical examination, the image quality of the fundus image is significant for clinical diagnosis by ophthalmologists and computer-aided diagnosis. However, the fundus image suffers from degradation due to improper clinical operation, environmental interference, and pathological factors. Despite enhancing the image quality decently, these existing methods based on deep learning perform well disappointingly in clinical fundus images because of the difficulty in collecting low-quality fundus images, the uncertainty of noise, and the complexity of fundus structures. To improve the generalization performance in low-quality fundus image enhancement, this paper proposes generalizable algorithms for low-quality fundus images. The main contributions of this research are listed below:

(1)  A network to annotation-freely restore cataractous fundus images (ArcNet) is proposed to boost the clinical practicability of enhancement for fundus images. Firstly, a cataract imaging model for low-quality fundus images is developed to stimulate the imaging pattern of cataracts. Then, the enhancement model is learned from the synthesized images and adapted to authentic cataract images by unsupervised domain adaptation. Annotations are unnecessary in ArcNet, where the high-frequency component is extracted from fundus images to replace segmentation in the preservation of retinal structures. Favorable performance is achieved using ArcNet against state-of-the-art algorithms, and the diagnosis of ocular fundus diseases in cataract patients is promoted by ArcNet.

(2) A domain generalization algorithm is designed for enhancing cataractous images (CataractDG) without paired or annotated data to boost the clinical enhancement. Firstly, domain randomization is adopted to simulate low-quality cataract images. Then, domain generalization is applied to learn domain-invariant features from synthesized data by extracting high-frequency components to conduct domain alignment. The capability of properly enhancing the quality and structure of fundus images without any annotated or target data promises the proposed algorithm outstanding clinical practicability.

(3) A generalizable pyramid constraint network is proposed for image enhancement (PCE-Net). Firstly, by analyzing and applying the interference factors of low-quality fundus images, high-quality images are randomly degraded to form sequences of low-quality ones sharing the same content. Then, individual low-quality images are decomposed to Laplacian pyramid features as the multi-level input for the enhancement. Finally, a feature pyramid constraint for the sequence is introduced to enforce the network to learn a degradation-invariant model and conduct generalizable enhancement.

These algorithms are improved gradually, starting from image enhancement of cataract fundus images using domain adaptation for clinical data, to image enhancement without requiring target domain data using domain generalization for clinical cataract images, and finally to image enhancement based on feature pyramid consistency for generalization to unseen domains. These proposed algorithms have been tested on clinical data and the experimental results have shown that the algorithms outperform the state-of-the-art methods, demonstrating the generalization ability in enhancement and effectiveness in downstream clinical tasks.

[1] ZHANG X Q, HU Y, XIAO Z J, et al. Machine learning for cataract classification/grading on ophthalmic imaging modalities: a survey[J]. Machine Intelligence Research, 2022, 19(3): 184-208.
[2] ZHANG X, XIAO Z, HIGASHITA R, et al. Adaptive feature squeeze network for nuclear cataract classification in AS-OCT image[J]. Journal of Biomedical Informatics, 2022, 128: 104037.
[3] BERNARDES R, SERRANHO P, LOBO C. Digital ocular fundus imaging: a review[J]. Ophthalmologica, 2011, 226(4): 161-181.
[4] SENGUPTA S, SINGH A, LEOPOLD H A, et al. Ophthalmic diagnosis using deep learning with fundus images–a critical review[J]. Artificial Intelligence in Medicine, 2020, 102: 101758.
[5] WANG S, JIN K, LU H, et al. Human visual system-based fundus image quality assessment of portable fundus camera photographs[J]. IEEE Transactions on Medical Imaging, 2015, 35(4): 1046-1055.
[6] LI T, BO W, HU C, et al. Applications of deep learning in fundus images: a review[J]. Medical Image Analysis, 2021: 101971.
[7] ZAGO G T, ANDREAO R V, DORIZZI B, et al. Retinal image quality assessment using deep learning[J]. Computers in Biology and Medicine, 2018, 103: 64-70.
[8] 赵连凯. 中老年白内障合并眼底病临床分析[J]. 当代医学, 2015, 21(19): 72-73.
[9] 赵家良. 我国眼健康事业的回顾与展望[J]. 中华眼科杂志, 2018, 54(8): 561-564.
[10] SHEN Z, FU H, SHEN J, et al. Modeling and enhancing low-quality retinal fundus images[J]. IEEE Transactions on Medical Imaging, 2020, 40(3): 996-1006.
[11] RAJ A, SHAH N A, TIWARI A K, et al. Multivariate regression-based convolutional neural network model for fundus image quality assessment[J]. IEEE Access, 2020, 8: 57810-57821.
[12] MUDDAMSETTY S M, MOESLUND T B. Multi-level quality assessment of retinal fundus images using deep convolution neural networks[C]//16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Application 2021. SCITEPRESS Digital Library, 2021: 661-668.
[13] XU Z, ZOU B, LIU Q. A deep retinal image quality assessment network with salient structure priors[J]. Multimedia Tools and Applications, 2023: 1-24.
[14] FU H, WANG B, SHEN J, et al. Evaluation of retinal image quality assessment networks in different color-spaces[C]//International Conference on Medical Image Computing and Computer Assisted Intervention. Springer, 2019: 48-56.
[15] PRATAP T, KOKIL P. Computer-aided diagnosis of cataract using deep transfer learning[J]. Biomedical Signal Processing and Control, 2019, 53: 101533.
[16] ZHANG S, WEBERS C A, BERENDSCHOT T T. Luminosity rectified blind Richardson Lucy deconvolution for single retinal image restoration[J]. Computer Methods and Programs in Biomedicine, 2023, 229: 107297.
[17] SAHU S, SINGH A K, GHRERA S, et al. An approach for de-noising and contrast enhancement of retinal fundus image using CLAHE[J]. Optics & Laser Technology, 2019, 110: 87-98.
[18] XIONG L, LI H, XU L. An enhancement method for color retinal images based on image formation model[J]. Computer Methods and Programs in Biomedicine, 2017, 143: 137-150.
[19] CAO L, LI H, ZHANG Y. Retinal image enhancement using low-pass filtering and 𝛼-rooting[J]. Signal Processing, 2020, 170: 107445.
[20] ISOLA P, ZHU J Y, ZHOU T, et al. Image-to-image translation with conditional adversarial networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017: 1125-1134.
[21] LI C, FU H, CONG R, et al. Nui-go: Recursive non-local encoder-decoder network for retinal image non-uniform illumination removal[C]//Proceedings of the 28th ACM International Conference on Multimedia. 2020: 1478-1487.
[22] ZHU J Y, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]//Proceedings of the IEEE International Conference on Computer Vision. 2017: 2223-2232.
[23] YOU Q, WAN C, SUN J, et al. Fundus image enhancement method based on CycleGAN[C]//2019 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE, 2019: 4500-4503.
[24] ZHAO H, YANG B, CAO L, et al. Data-driven enhancement of blurry retinal images via generative adversarial networks[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2019: 75-83.
[25] YOO T K, CHOI J Y, KIM H K. CycleGAN-based deep learning technique for artifact reduction in fundus photography[J]. Graefe’s Archive for Clinical and Experimental Ophthalmology, 2020, 258(8): 1631-1637.
[26] MA Y, LIU J, LIU Y, et al. Structure and illumination constrained GAN for medical image enhancement[J]. IEEE Transactions on Medical Imaging, 2021, 40(12): 3955-3967.
[27] PARK T, EFROS A A, ZHANG R, et al. Contrastive learning for unpaired image-to-image translation[C]//European Conference on Computer Vision. Springer, 2020: 319-345.
[28] CHENG P, LIN L, HUANG Y, et al. Prior guided fundus image quality enhancement via contrastive learning[C]//2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI). IEEE, 2021: 521-525.
[29] CHENG P, LIN L, HUANG Y, et al. I-secret: importance-guided fundus image enhancement via semi-supervised contrastive constraining[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2021: 87-96.
[30] PELI E, PELI T. Restoration of retinal images obtained through cataracts[J]. IEEE Transactions on Medical Imaging, 1989, 8(4): 401-406.
[31] MITRA A, ROY S, ROY S, et al. Enhancement and restoration of non-uniform illuminated fundus image of retina obtained through thin layer of cataract[J]. Computer Methods and Programs in Biomedicine, 2018, 156: 169-178.
[32] CHENG J, LI Z, GU Z, et al. Structure-preserving guided retinal image filtering and its application for optic disk analysis[J]. IEEE Transactions on Medical Imaging, 2018, 37(11): 2536-2546.
[33] LUO Y, CHEN K, LIU L, et al. Dehaze of cataractous retinal images using an unpaired generative adversarial network[J]. IEEE Journal of Biomedical and Health Informatics, 2020, 24(12): 3374-3383.
[34] FORACCHIA M, GRISAN E, RUGGERI A. Luminosity and contrast normalization in retinal images[J]. Medical Image Analysis, 2005, 9(3): 179-190.
[35] PIZER S M, AMBURN E P, AUSTIN J D, et al. Adaptive histogram equalization and its variations[J]. Computer Vision, Graphics, and Image Processing, 1987, 39(3): 355-368.
[36] SETIAWAN A W, MENGKO T R, SANTOSO O S, et al. Color retinal image enhancement using CLAHE[C]//International Conference on ICT for Smart Society. IEEE, 2013: 1-3.
[37] HE K, SUN J, TANG X. Single image haze removal using dark channel prior[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 33(12): 2341-2353.
[38] PÉREZ A D, PERDOMO O, RIOS H, et al. A Conditional Generative Adversarial Network Based Method for Eye Fundus Image Quality Enhancement[C]//International Workshop on Ophthalmic Medical Image Analysis. Springer, 2020: 185-194.
[39] ZHOU K, LIU Z, QIAO Y, et al. Domain generalization: A survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(4): 4396-4415.
[40] BAKER M L, HAND P J, WANG J J, et al. Retinal signs and stroke: revisiting the link between the eye and brain[J]. Stroke, 2008, 39(4): 1371-1379.
[41] ABRÀMOFF M D, GARVIN M K, SONKA M. Retinal imaging and image analysis[J]. IEEE Reviews in Biomedical Engineering, 2010, 3: 169-208.
[42] QI Y, YANG Z, SUN W, et al. A comprehensive overview of image enhancement techniques [J]. Archives of Computational Methods in Engineering, 2021: 1-25.
[43] MAHESH B. Machine learning algorithms-a review[J]. International Journal of Science and Research (IJSR), 2020, 9: 381-386.
[44] PANG Y, LIN J, QIN T, et al. Image-to-image translation: Methods and applications[J]. IEEE Transactions on Multimedia, 2021, 24: 3859-3881.
[45] PAN S J, YANG Q. A survey on transfer learning[J]. IEEE Transactions on Knowledge and Data Engineering, 2010, 22(10): 1345-1359.
[46] ZHUANG F, QI Z, DUAN K, et al. A comprehensive survey on transfer learning[J]. Proceedings of the IEEE, 2020, 109(1): 43-76.
[47] KORA P, OOI C P, FAUST O, et al. Transfer learning techniques for medical image analysis: A review[J]. Biocybernetics and Biomedical Engineering, 2022, 42(1): 79-107.
[48] ZHOU K, HE W, XU Y, et al. Feature selection and transfer learning for Alzheimer’ s disease clinical diagnosis[J]. Applied Sciences, 2018, 8(8): 1372.
[49] KARIMI D, WARFIELD S K, GHOLIPOUR A. Transfer learning in medical image segmentation: New insights from analysis of the dynamics of model parameters and learned representations[J]. Artificial Intelligence in Medicine, 2021, 116: 102078.
[50] KOUW W M, LOOG M. A review of domain adaptation without target labels[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 43(3): 766-785.
[51] PAN S J, TSANG I W, KWOK J T, et al. Domain adaptation via transfer component analysis [J]. IEEE Transactions on Neural Networks, 2010, 22(2): 199-210.
[52] CRESWELL A, WHITE T, DUMOULIN V, et al. Generative adversarial networks: an overview [J]. IEEE Signal Processing Magazine, 2018, 35(1): 53-65.
[53] GHIFARY M, KLEIJN W B, ZHANG M, et al. Deep reconstruction-classification networks for unsupervised domain adaptation[C]//Proceedings of the 14th European Conference on Computer Vision. Springer, 2016: 597-613.
[54] YANG Y, SOATTO S. Fda: Fourier domain adaptation for semantic segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 4085-4095.
[55] GHIFARY M, KLEIJN W B, ZHANG M, et al. Domain generalization for object recognition with multi-task autoencoders[C]//Proceedings of the IEEE International Conference on Computer Vision. 2015: 2551-2559.
[56] LIU Q, DOU Q, HENG P A. Shape-aware meta-learning for generalizing prostate MRI segmentation to unseen domains[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2020: 475-485.
[57] HUANG J, GUAN D, XIAO A, et al. Fsdr: Frequency space domain randomization for domain generalization[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 6891-6902.
[58] FLAXMAN S R, BOURNE R R, RESNIKOFF S, et al. Global causes of blindness and distance vision impairment 1990–2020: a systematic review and meta-analysis[J]. The Lancet Global Health, 2017, 5(12): e1221-e1234.
[59] LEE C M, AFSHARI N A. The global state of cataract blindness[J]. Current Opinion in Ophthalmology, 2017, 28(1): 98-103.
[60] SIDDIQUI A A, LADAS J G, LEE J K. Artificial intelligence in cornea, refractive, and cataract surgery[J]. Current Opinion in Ophthalmology, 2020, 31(4): 253-260.
[61] KUMAR B V, PHILLIPS R P, PRASAD S. Multifocal intraocular lenses in the setting of glaucoma[J]. Current Opinion in Ophthalmology, 2007, 18(1): 62-66.
[62] FLESNER P, SANDER B, HENNING V, et al. Cataract surgery on diabetic patients. A prospective evaluation of risk factors and complications[J]. Acta Ophthalmologica Scandinavica, 2002, 80(1): 19-24.
[63] KIM S J, EQUI R, BRESSLER N M. Analysis of macular edema after cataract surgery in patients with diabetes using optical coherence tomography[J]. Ophthalmology, 2007, 114(5): 881-889.
[64] ALIO J L, PLAZA-PUCHE A B, FÉRNANDEZ-BUENAGA R, et al. Multifocal intraocular lenses: an overview[J]. Survey of Ophthalmology, 2017, 62(5): 611-634.
[65] PELI E, SCHWARTZ B. Enhancement of fundus photographs taken through cataracts[J]. Ophthalmology, 1987, 94: 10-13.
[66] HE K, SUN J, TANG X. Guided image filtering[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 35(6): 1397-1409.
[67] DELORI F C, GRAGOUDAS E S, FRANCISCO R, et al. Monochromatic ophthalmoscopy and fundus photography: the normal fundus[J]. Archives of ophthalmology, 1977, 95(5): 861-868.
[68] LAND E H. The retinex theory of color vision[J]. Scientific American, 1977, 237(6): 108-129.
[69] GETREUER P. A survey of Gaussian convolution algorithms[J]. Image Processing On Line, 2013, 2013: 286-310.
[70] GUO J, PANG Z, YANG F, et al. Study on the method of fundus image generation based on improved GAN[J]. Mathematical Problems in Engineering, 2020, 2020: 1-13.
[71] LIU Y, WANG X, WANG L, et al. A modified leaky ReLU scheme (MLRS) for topology optimization with multiple materials[J]. Applied Mathematics and Computation, 2019, 352: 188-204.
[72] SANTURKAR S, TSIPRAS D, ILYAS A, et al. How does batch normalization help optimization?[J]. Advances in Neural Information Processing Systems, 2018, 31: 2488–2498.
[73] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016: 770-778.
[74] JENI L A, COHN J F, DE LA TORRE F. Facing imbalanced data–recommendations for the use of performance metrics[C]//Humaine Association Conference on Affective Computing and Intelligent Interaction. 2013: 245-251.
[75] WANG W, YAN W, FOTIS K, et al. Cataract surgical rate and socioeconomics: a global study[J]. Investigative Ophthalmology & Visual Science, 2016, 57(14): 5872-5881.
[76] PHAM T Q, WANG J J, ROCHTCHINA E, et al. Systemic and ocular comorbidity of cataract surgical patients in a western Sydney public hospital[J]. Clinical & Experimental Ophthalmology, 2004, 32(4): 383-387.
[77] ZHOU K, YANG Y, QIAO Y, et al. Domain adaptive ensemble learning[J]. IEEE Transactions on Image Processing, 2021, 30: 8008-8018.
[78] KIM M, BYUN H. Learning texture invariant representation for domain adaptation of semantic segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 12975-12984.
[79] TREMBLAY J, PRAKASH A, ACUNA D, et al. Training deep networks with synthetic data: Bridging the reality gap by domain randomization[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2018: 969-977.
[80] HORE A, ZIOU D. Image quality metrics: PSNR vs. SSIM[C]//2010 20th International Conference on Pattern Recognition. IEEE, 2010: 2366-2369.
[81] JADON S. A survey of loss functions for semantic segmentation[C]//2020 IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB). IEEE, 2020: 1-7.
[82] LI H, LIU H, HU Y, et al. Restoration of cataract fundus images via unsupervised domain adaptation[C]//2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI). IEEE, 2021: 516-520.
[83] RONNEBERGER O, FISCHER P, BROX T. U-net: Convolutional networks for biomedical image segmentation[C]//International Conference on Medical Image Computing and Computer Assisted Intervention. Springer, 2015: 234-241.
[84] DENTON E, CHINTALA S, SZLAM A, et al. Deep generative image models using a Laplacian pyramid of adversarial networks[J]. Advances in Neural Information Processing Systems, 2015: 1486-1494.
[85] LIANG J, ZENG H, ZHANG L. High-resolution photorealistic image translation in real-time: a laplacian pyramid translation network[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 9392-9400.
[86] GHIASI G, FOWLKES C C. Laplacian pyramid reconstruction and refinement for semantic segmentation[C]//European Conference on Computer Vision. Springer, 2016: 519-534.
[87] LI X, HU X, QI X, et al. Rotation-oriented collaborative self-supervised learning for retinal disease diagnosis[J]. IEEE Transactions on Medical Imaging, 2021, 40(9): 2284-2294.
[88] HE K, ZHANG X, REN S, et al. Spatial pyramid pooling in deep convolutional networks for visual recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(9): 1904-1916.