针对白内障手术视频的语义分割算法研究

白内障是全球首要致盲眼病，目前最有效的治疗方法是白内障手术。由于眼球结构复杂，手术过程中需要非常小心，以避免损伤其他眼部组织。计算机辅助白内障手术不仅能减轻医生的工作量和压力，而且能降低手术意外的风险。其中，白内障手术视频的语义分割是一个重要研究方向，它可以在像素位置上识别出手术器械或器官组织的类别。精确的分割结果不仅可以直接用于监测手术并为医生提供反馈和指导，而且为手术阶段识别和器械精确跟踪等任务奠定了基础。

近年来，学术界已经涌现出大量视频语义分割算法。然而，现有算法在捕捉手术视频中的局部细节方面存在不足，难以直接适配复杂的手术场景。此外，由于对手术视频进行标注需要大量的人力、物力和医疗资源，因而实际场景中往往只有少量的标注视频帧，标注数据的稀缺限制了有监督模型的性能。因此，本文针对上述两类问题，分别设计有监督和半监督的视频语义分割算法，主要研究内容总结如下：

针对有监督场景中现有视频语义分割方法在捕捉局部细节方面存在不足的问题，本文提出了一种基于视频全局和局部信息融合的视频语义分割方法。首先设计基于注意力的全局上下文模块来提取视频帧的全局上下文特征，同时利用动态卷积构建局部上下文模块来提取视频帧的局部上下文特征，然后通过设计多尺度融合模块实现了全局与局部特征在不同尺度上的有效整合。在两个公开的手术视频数据集上的实验结果证明了该方法能够同时提取全局丰富的特征并捕捉到关键的局部细节从而提高分割精度。基于同样的主干网络，该方法在平均交并比指标上比图像语义分割方法高约3%，比其他视频语义分割方法高约2%。

针对半监督场景中现有方法使用单帧图像生成的伪标签不可靠的问题，本文提出了一种基于帧间特征转换的半监督视频语义分割方法。该方法在连续帧中利用有标签帧的类特征原型对无标签帧的特征进行转换，然后设计交叉伪标签损失和特征对比损失作为无标签帧的约束；此外，使用无标签帧的类特征原型对有标签帧的特征进行转换，并使用真实标签进行监督。在公开的白内障手术数据集上的实验结果证明了该方法在标注样本稀缺时对分割精度的显著提升。仅使用100张有标签视频帧时，该方法的平均交并比指标相较于监督方法提升约3%。

The most common cause of blindness worldwide is cataracts. Currently, the most effective treatment is cataract surgery. Due to the complex structure of the eyeball, doctors need to perform cataract surgery very carefully to avoid damaging other eye tissues. Computer assisted cataract surgery not only reduces the workload and stress of doctors, but also reduces the risk of surgical accidents. Semantic segmentation of cataract surgery videos is an important research direction, which identifies the categories of surgical instruments or organ tissues at pixel locations. Accurate segmentation results can not only be directly used to monitor surgeries and provide guidance to doctors, but also lay the foundation for other tasks such as surgical stage identification and accurate tracking of instruments.

In recent years, many video semantic segmentation algorithms have been proposed in the academic community. However, the existing algorithms have shortcomings in capturing local details in surgical videos, making it difficult to adapt to complex surgical scenes directly. In addition, since annotating surgical videos requires a large amount of manpower and medical resources, there are only a small number of annotated videos in actual scenarios. The scarcity of annotated data limits the performance of supervised models. Therefore, we design supervised and semi-supervised video semantic segmentation algorithms to address the above problems. The main research contents are summarized as follows.

Aiming at the problem that existing video semantic segmentation methods in supervised scenes are insufficient in capturing local details, a video semantic segmentation method is proposed based on the fusion of global and local information in the video in this thesis. First, an attention-based global context module is designed to extract the global context features of the video frames. At the same time, dynamic convolution is used to build a local context module to extract the local context features of the video frames. Then, the multi-scale fusion module is designed to realize the integration of global and local features at different scales. Experimental results on two public surgery video datasets demonstrate that our method can extract global features and capture local details to improve segmentation accuracy. Based on the same backbone network, our method is about 3% higher than the image semantic segmentation method in terms of mean intersection and union, and our method is about 2% higher than other video semantic segmentation methods.

Aiming at the problem of unreliable pseudo labels generated by existing methods using a single image in semi-supervised scenarios, a semi-supervised video semantic segmentation method with cross-supervision in consecutive frames is proposed in this thesis. The class feature prototypes of labeled frames are used to convert the features of unlabeled frames, and we propose a cross pseudo-label loss and feature contrastive loss as constraints for the unlabeled frames. In addition, the class feature prototypes of the unlabeled frames are used to convert the features of labeled frames, and we utilize ground truth to supervise them. Experimental results on a public cataract surgery dataset demonstrate that our method significantly improves segmentation accuracy when labeled samples are scarce. When only 100 labeled video frames are used, the mean intersection over the union of our method is improved by about 3% compared to the supervised method.

[1] 赵家良. 我国眼健康事业的回顾与展望[J]. 中华眼科杂志, 2018, 54(8): 561-564.
[2] Flaxman S R, Bourne R R A, Resnikoff S, et al. Global causes of blindness and distance vision impairment 1990–2020: a systematic review and meta-analysis[J]. The Lancet Global Health, 2017, 5(12): 1221-1234.
[3] 王倩倩, 章涛, 李傅冬, 等. 老年人群白内障的影响因素分析[J]. 预防医学, 2023, 35(4): 311-315.
[4] 俞方良, 易昀敏, 兰绪达, 等. 微切口白内障超声乳化术的临床研究[J]. 中国实用眼科杂志, 2010, 28(1): 25-27.
[5] Abell R G, Kerr N M, Vote B J. Femtosecond laser-assisted cataract surgery compared with conventional cataract surgery[J]. Clinical and Experimental Ophthalmology, 2013, 41(5): 455-462.
[6] Zisimopoulos O, Flouty E, Luengo I, et al. Deepphase: Surgical phase recognition in cataracts videos[C]//Medical Image Computing and Computer Assisted Intervention. 2018: 265-272.
[7] Ginesi M, Meli D, Roberti A, et al. Autonomous task planning and situation awareness in robotic surgery[C]//IEEE/RSJ International Conference on Intelligent Robots and Systems. 2020: 3144-3150.
[8] Du X, Allan M, Dore A, et al. Combined 2D and 3D tracking of surgical instruments for minimally invasive and robotic-assisted surgery[J]. International Journal of Computer Assisted Radiology and Surgery, 2016, 11(1): 1109-1119.
[9] Zhao Z, Chen Z, Voros S, et al. Real-time tracking of surgical instruments based on spatio-temporal context and deep learning[J]. Computer Assisted Surgery, 2019, 24(1): 20-29.
[10] Chen Z, Zhao Z, Cheng X. Surgical instruments tracking based on deep learning with lines detection and spatio-temporal context[C]//Chinese Automation Congress. 2017: 2711-2714.
[11] Kurmann T, Marquez Neila P, Du X, et al. Simultaneous recognition and pose estimation of instruments in minimally invasive surgery[C]//Medical Image Computing and Computer Assisted Intervention. 2017: 505-513.
[12] Du X, Kurmann T, Chang P L, et al. Articulated multi-instrument 2D pose estimation using fully convolutional networks[J]. IEEE Transactions on Medical Imaging, 2018, 37(5): 1276-1287.
[13] Colleoni E, Moccia S, Du X, et al. Deep learning based robotic tool detection and articulation estimation with spatio-temporal layers[J]. IEEE Robotics and Automation Letters, 2019, 4(3): 2714-2721.
[14] Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segment-ation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015: 3431-3440.
[15] Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation[C]//Medical Image Computing and Computer Assisted Intervention. 2015: 234-241.
[16] Zhao H, Shi J, Qi X, et al. Pyramid scene parsing network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017: 2881-2890.
[17] Shvets A A, Rakhlin A, Kalinin A A, et al. Automatic instrument segmentation in robot-assisted surgery using deep learning[C]//International Conference on Machine Learning and Applications. 2018: 624-628.
[18] Hasan S M K, Linte C A. U-NetPlus: A modified encoder-decoder U-Net architecture for semantic and instance segmentation of surgical instruments from laparoscopic images[C]//International Conference of the IEEE Engineering in Medicine and Biology Society. 2019: 7205-7211.
[19] Yu L, Wang P, Yu X, et al. A holistically-nested U-net: Surgical instrument segmentation based on convolutional neural network[J]. Journal of Digital Imaging, 2020, 33(2): 341-347.
[20] Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C]//Advances in Neural Information Processing Systems. 2017: 5998-6008.
[21] Yang L, Wang H, Gu Y, et al. TMA-Net: A transformer-based multi-scale attention network for surgical instrument segmentation[J]. IEEE Transactions on Medical Robotics and Bionics, 2023, 5(2): 323-334.
[22] Yang L, Wang H, Bian G, et al. HCTA-Net: A hybrid CNN-transformer attention network for surgical instrument segmentation[J]. IEEE Transactions on Medical Robotics and Bionics, 2023, 5(4): 929-944.
[23] Hassan C S, Ridzuan A. Parallel cross window attention transformer and CNN model for segmentation of instrument during surgery[C]//International Conference on Artificial Life and Robotics. 2024: 204-208.
[24] Kirillov A, Mintun E, Ravi N, et al. Segment anything[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 4015-4026.
[25] Meng H, Chen L, Zhu S, et al. Zero-shot kidney stone segmentation based on segmentation anything model for robotic-assisted endoscope navigation[C]//International Conference on Intelligent Robotics and Applications. 2023: 80-90.
[26] Zhou Z, Alabi T, Wei O, et al. Text promptable surgical instrument segmentation with vision-language models[C]//Advances in Neural Information Processing Systems. 2023: 604-624.
[27] García-Peraza-Herrera L C, Li W, Gruijthuijsen C, et al. Real-time segmentation of non-rigid surgical tools based on deep learning and tracking[C]//International Workshop on Computer Assisted and Robotic Endoscopy. 2016: 84-95.
[28] Islam M, Atputharuban D A, Ramesh R, et al. Real-time instrument segmentation in robotic surgery using auxiliary supervised deep adversarial learning[J]. IEEE Robotics and Automation Letters, 2019, 4(2): 2188-2195.
[29] Li Y, Shi J, Lin D. Low-latency video semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 5997-6005.
[30] Mishra K, Sathish R, Sheet D. Learning latent temporal connectionism of deep residual visual abstractions for identifying surgical tools in laparoscopy procedures[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2017: 58-65.
[31] Grammatikopoulou M, Sanchez-Matilla R, Bragman F, et al. A spatio-temporal network for video semantic segmentation in surgical videos[J]. International Journal of Computer Assisted Radiology and Surgery, 2024, 19(2): 375-382.
[32] Wang J, Jin Y, Wang L, et al. Efficient global-local memory for real-time instrument segmentation of robotic surgical video[C]//Medical Image Computing and Computer Assisted Intervention. 2021: 341-351.
[33] Jin Y, Yu Y, Chen C, et al. Exploring intra-and inter-video relation for surgical semantic scene segmentation[J]. IEEE Transactions on Medical Imaging, 2022, 41(11): 2991-3002.
[34] Yao H, Hu X, Li X. Enhancing pseudo label quality for semi-supervised domain-generalized medical image segmentation[C]//Proceedings of the AAAI conference on artificial intelligence. 2022, 36(3): 3099-3107.
[35] Wang X, Yuan Y, Guo D, et al. SSA-Net: Spatial self-attention network for COVID-19 pneumonia infection segmentation with semi-supervised few-shot learning[J]. Medical Image Analysis, 2022, 79(1): 1024-1029.
[36] Zhang Z, Tian C, Bai H X, et al. Discriminative error prediction network for semi-supervised colon gland segmentation[J]. Medical Image Analysis, 2022, 79(1): 956-968.
[37] Shi Y, Zhang J, Ling T, et al. Inconsistency-aware uncertainty estimation for semi-supervised medical image segmentation[J]. IEEE Transactions on Medical Imaging, 2021, 41(3): 608-620.
[38] Huang W, Chen C, Xiong Z, et al. Semi-supervised neuron segmentation via reinforced consistency learning[J]. IEEE Transactions on Medical Imaging, 2022, 41(11): 3016-3028.
[39] Chen X, Zhou H Y, Liu F, et al. MASS: Modality-collaborative semi-supervised segmentation by exploiting cross-modal consistency from unpaired CT and MRI images[J]. Medical Image Analysis, 2022, 79(1): 1025-1036.
[40] Zhang Y, Yang L, Chen J, et al. Deep adversarial networks for biomedical image segmentation utilizing unannotated images[C]//Medical Image Computing and Computer Assisted Intervention. 2017: 408-416.
[41] Wu Y, Xu M, Ge Z, et al. Semi-supervised left atrium segmentation with mutual consistency training[C]//Medical Image Computing and Computer Assisted Intervention. 2021: 297-306.
[42] Wu Z, Lau C Y, Zhou Q, et al. Surgivisor: Transformer-based semi-supervised instrument segmentation for endoscopic surgery[J]. Biomedical Signal Processing and Control, 2024, 87(1): 484-494.
[43] Yuan Z, Lin J, Zhang D. Hierarchical Semi-Supervised Learning Framework for Surgical Gesture Segmentation and Recognition Based on Multi-Modality Data[C]//IEEE/RSJ International Conference on Intelligent Robots and Systems. 2023: 7659-7666.
[44] Lou A, Tawfik K, Yao X, et al. Min-max similarity: A contrastive semi-supervised deep learning network for surgical tools segmentation[J]. IEEE Transactions on Medical Imaging, 2023, 42(10): 2832-2841.
[45] He Z, Qiu J. Surgical Instrument Segmentation Based on Geometric-aware Semi-supervised Contrastive Learning[C]//International Conference on Cloud Computing and Intelligent Systems. 2023: 393-397.
[46] Krizhevsky A, Sutskever I, Hinton G E. Imagenet classification with deep convolutional neural networks[C]//Advances in Neural Information Processing Systems. 2012: 1097-1105.
[47] Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16x16 words: Transformers for image recognition at scale[C]//International Conference on Learning Representations. 2021: 1-21.
[48] Xie E, Wang W, Yu Z, et al. SegFormer: Simple and efficient design for semantic segmentation with transformers[C]//Advances in Neural Information Processing Systems. 2021: 12077-12090.
[49] Liu Z, Lin Y, Cao Y, et al. Swin transformer: Hierarchical vision transformer using shifted windows[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 10012-10022.
[50] Zheng S, Lu J, Zhao H, et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2021: 6881-6890.
[51] Lee Y, Kim J, Willette J, et al. Mpvit: Multi-path vision transformer for dense prediction[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2022: 7287-7296.
[52] He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016: 770-778.
[53] Zhu X, Xiong Y, Dai J, et al. Deep feature flow for video recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017: 2349-2358.
[54] Li J, Wang W, Chen J, et al. Video semantic segmentation via sparse temporal transformer[C]//Proceedings of the ACM International Conference on Multimedia. 2021: 59-68.
[55] An S, Liao Q, Lu Z, et al. Dual correlation network for efficient video semantic segmentation[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 34(3): 1572-1585.
[56] Sun G, Liu Y, Ding H, et al. Coarse-to-fine feature mining for video semantic segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 3126-3137.
[57] P, Caba F, Wang O, et al. Temporally distributed networks for fast video semantic segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 8818-8827.
[58] Bair E. Semi-supervised clustering methods[J]. Wiley Interdisciplinary Reviews: Computational Statistics, 2013, 5(5): 349-361.
[59] Shen W, Peng Z, Wang X, et al. A survey on label-efficient deep image segmentation: Bridging the gap between weak supervision and dense prediction[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(8): 9284-9305.
[60] Lee D H. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks[C]//International Conference on Machine Learning Workshops. 2013, 3(2): 896-912.
[61] Ke Z, Qiu D, Li K, et al. Guided collaborative training for pixel-wise semi-supervised learning[C]//Proceedings of the European Conference on Computer Vision. 2020: 429-445.
[62] French G, Aila T, Laine S, et al. Semi-supervised semantic segmentation needs strong, varied perturbations[C]//British Machine Vision Conference. 2020: 1-14.
[63] Zou Y, Zhang Z, Zhang H, et al. Pseudoseg: Designing pseudo labels for se-mantic segmentation [C]//International Conference on Learning Representations. 2021: 1-12.
[64] Chen X, Yuan Y, Zeng G, et al. Semi-supervised semantic segmentation with cross pseudo supervision[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 2613-2622.
[65] Gadde R, Jampani V, Gehler P V. Semantic video cnns through representation warping[C]//Proceedings of the IEEE International Conference on Computer Vision. 2017: 4453-4462.
[66] Nilsson D, Sminchisescu C. Semantic video segmentation by gated recurrent flow propagation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 6819-6828.
[67] Paul M, Danelljan M, Van Gool L, et al. Local memory attention for fast video semantic segmentation[C]//IEEE/RSJ International Conference on Intelligent Robots and Systems. 2021: 1102-1109.
[68] Sun G, Liu Y, Tang H, et al. Mining relations among cross-frame affinities for video semantic segmentation[C]//Proceedings of the European Conference on Computer Vision. 2022: 522-539.
[69] Lin B, Zhang S, Yu X. Gait recognition via effective global-local feature representation and local temporal aggregation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 14648-14656.
[70] Su Z, Yang Y, Huang S, et al. CTCP: Cross transformer and CNN for pansharpening[C]//Proceedings of the ACM International Conference on Multimedia. 2023: 3003-3011.
[71] Fu J, Liu J, Tian H, et al. Dual attention network for scene segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019: 3146-3154.
[72] Zhou Z, Ren L, Xiong P, et al. Enhanced memory network for video segmentation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. 2019: 689-692.
[73] Chen Y, Dai X, Liu M, et al. Dynamic convolution: Attention over convolution kernels[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2020: 11030-11039.
[74] Grammatikopoulou M, Flouty E, Kadkhodamohammadi A, et al. CaDIS: Cataract dataset for surgical RGB-image segmentation[J]. Medical Image Analysis, 2021, 71(1): 747-755.
[75] Allan M, Kondo S, Bodenstedt S, et al. 2018 robotic scene segmentation challenge[J/OL]. 2020. arXiv:2001.11190. https://arxiv.org/abs/2001.11190.
[76] Chen L C, Zhu Y, Papandreou G, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation[C]//Proceedings of the European Conference on Computer Vision. 2018: 801-818.
[77] Xiao T, Liu Y, Zhou B, et al. Unified perceptual parsing for scene understanding[C]//Proceedings of the European Conference on Computer Vision. 2018: 418-434.
[78] Pissas T, Ravasio C S, Da Cruz L, et al. Effective semantic segmentation in cataract surgery: What matters most?[C]//Medical Image Computing and Computer Assisted Intervention. 2021: 509-518.
[79] Wang J, Sun K, Cheng T, et al. Deep high-resolution representation learning for visual recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 43(10): 3349-3364.
[80] Jin Y, Cheng K, Dou Q, et al. Incorporating temporal prior from motion flow for instrument segmentation in minimally invasive surgery video[C]//Medical Image Computing and Computer Assisted Intervention. 2019: 440-448.
[81] Jain S, Wang X, Gonzalez J E. Accel: A corrective fusion network for efficient semantic segmentation on video[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 8866-8875.
[82] Chu X, Tian Z, Wang Y, et al. Twins: Revisiting the design of spatial attention in vision transformers[C]//Advances in Neural Information Processing Systems. 2021: 9355-9366.
[83] Chollet F. Xception: Deep learning with depthwise separable convolutions[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017: 1251-1258.
[84] Han Q, Fan Z, Dai Q, et al. On the connection between local attention and dynamic depth-wise convolution[C]//International Conference on Learning Representations. 2022: 1-14.
[85] Chen L C, Lopes R G, Cheng B, et al. Naive-student: Leveraging semi-supervised learning in video sequences for urban scene segmentation[C]//Proceedings of the European Conference on Computer Vision. 2020: 695-714.
[86] Mendel R, De Souza L A, Rauber D, et al. Semi-supervised segmentation based on error-correcting supervision[C]//Proceedings of the European Conference on Computer Vision. 2020: 141-157.
[87] Ibrahim M S, Vahdat A, Ranjbar M, et al. Semi-supervised semantic image segmentation with self-correcting networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2020: 12715-12725.
[88] Ke Z, Wang D, Yan Q, et al. Dual student: Breaking the limits of the teacher in semi-supervised learning[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019: 6728-6736.
[89] Lai X, Tian Z, Jiang L, et al. Semi-supervised semantic segmentation with directional context-aware consistency[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 1205-1214.
[90] Xie Q, Luong M T, Hovy E, et al. Self-training with noisy student improves imagenet classification[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2020: 10687-10698.
[91] Yang L, Zhuo W, Qi L, et al. St++: Make self-training work better for semi-supervised semantic segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 4268-4277.
[92] Arazo E, Ortego D, Albert P, et al. Pseudo-labeling and confirmation bias in deep semi-supervised learning[C]//International Joint Conference on Neural Networks. 2020: 1-8.
[93] Sohn K, Berthelot D, Carlini N, et al. Fixmatch: Simplifying semi-supervised learning with consistency and confidence[C]//Advances in Neural Information Processing Systems. 2020: 596-608.
[94] Al Hajj H, Lamard M, Conze P H, et al. CATARACTS: Challenge on automatic tool annotation for cataract surgery[J]. Medical Image Analysis, 2019, 52(1): 24-41.
[95] Ouali Y, Hudelot C, Tami M. Semi-supervised semantic segmentation with cross-consistency training[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 12674-12684.