基于跨模态模型预训练和质量增强的医学影像分析

医学影像分析在临床医学中扮演着至关重要的角色。然而，由于医学影像的 复杂性和多样性，智能自动化的医学影像分析任务往往难度较大。传统的机器学 习方法在医学影像分析中取得了一定的进展，但这些方法通常依赖于手工提取的 特征或单一模态的数据，限制了它们的表现和推广能力。近年来，深度学习技术 在医学影像分析领域得到了广泛应用。然而，一方面医学影像的数据获取难度大， 标注成本高，模型训练的最终质量往往不尽如人意，难以泛化至临床任务中；另一 方面，临床数据的质量往往参差不齐，模型难以在低质量数据上表现出鲁棒性能。 因此，本研究提出了一种基于模型和数据质量增强的通用医学影像分析框架，针 对医学影像中的模型泛化能力问题，联合临床诊断报告进行跨模态自监督预训练， 通过将临床报告中的基于影像的诊断先验知识引入到影像模块的编码中，大大提 高了少样本医学影像训练情景下的模型质量；另一方面，针对临床中的低质量图 像，本研究从成像学角度出发，引入基于半监督和对比学习的生成对抗网络，拉 近低质量图像和高质量图像的特征分布，实现对临床低质量图像的显著质量增强。 通过结合模型质量提升和数据质量提升算法，本研究在多个下游任务和领域内最 先进的算法进行了比较，针对以上两个挑战性问题，在多个任务和指标上取得了 领先的结果。

[1] BERGTHOLDT M, WIEMKER R, KLINDER T. Pulmonary nodule detection using a cascaded SVM classifier[C]//Medical Imaging 2016: Computer­Aided Diagnosis: volume 9785. SPIE, 2016: 268­278.
[2] KAMIYA Y, GE L, HONG T, et al. The nature of spin excitations in the one­third magnetization plateau phase of Ba3CoSb2O9 [J]. Nature communications, 2018, 9(1): 1­11.
[3] LI Y, CERITOGLU C, JIANG H, et al. BrainGPS: a cloud­based platform for neuroimage analysis and neuroradiological studies[C]//23th ISMRM Meeting (Toronto, ON). 2015.
[4] ESTEVA A, KUPREL B, NOVOA R A, et al. Dermatologist­level classification of skin cancer with deep neural networks[J]. nature, 2017, 542(7639): 115­118.
[5] GULSHAN V, PENG L, CORAM M, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs[J]. Jama, 2016, 316(22): 2402­2410.
[6] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16x16 words:Transformers for image recognition at scale[A]. 2020.
[7] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 770­778.
[8] SZEGEDY C, LIU W, JIA Y, et al. Going deeper with convolutions[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2015: 1­9.
[9] ZHAO H, JIA J, KOLTUN V. Exploring self­attention for image recognition[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 10076­10085.
[10] CUI Y, JIA M, LIN T Y, et al. Class­balanced loss based on effective number of samples[C]// Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019: 9268­9277.
[11] SUDRE C H, LI W, VERCAUTEREN T, et al. Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations[C]//Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: Third International Workshop, DLMIA 2017, and 7th International Workshop, ML­CDS 2017, Held in Conjunction with MICCAI 2017, Québec City, QC, Canada, September 14, Proceedings 3. Springer, 2017: 240­248.
[12] ZOPH B, VASUDEVAN V, SHLENS J, et al. Learning transferable architectures for calable image recognition[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 8697­8710.
[13] RONNEBERGER O, FISCHER P, BROX T. U­net: Convolutional networks for biomedical image segmentation[C]//International Conference on Medical image computing and computerassisted intervention. Springer, 2015: 234­241.
[14] REN S, HE K, GIRSHICK R, et al. Faster r­cnn: Towards real­time object detection with region proposal networks[J]. Advances in neural information processing systems, 2015, 28 ....... （详见论文附录）