ACT-Net: Anchor-Context Action Detection in Surgery Videos

Hao, Luoying1,2; Hu, Yan2; Lin, Wenjun2,3; Wang, Qun4; Li, Heng2; Fu, Huazhu5; Duan, Jinming1; Liu, Jiang2

10.1007/978-3-031-43996-4_19

2023

26th International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI)

0302-9743

1611-3349

978-3-031-43995-7

MEDICAL IMAGE COMPUTING AND COMPUTER ASSISTED INTERVENTION, MICCAI 2023, PT IX

14228

OCT 08-12, 2023

null,Vancouver,CANADA

GEWERBESTRASSE 11, CHAM, CH-6330, SWITZERLAND

SPRINGER INTERNATIONAL PUBLISHING AG

Recognition and localization of surgical detailed actions is an essential component of developing a context-aware decision support system. However, most existing detection algorithms fail to provide high-accuracy action classes even having their locations, as they do not consider the surgery procedure's regularity in the whole video. This limitation hinders their application. Moreover, implementing the predictions in clinical applications seriously needs to convey model confidence to earn entrustment, which is unexplored in surgical action prediction. In this paper, to accurately detect fine-grained actions that happen at every moment, we propose an anchor-context action detection network (ACT-Net), including an anchor-context detection (ACD) module and a class conditional diffusion (CCD) module, to answer the following questions: 1) where the actions happen; 2) what actions are; 3) how confidence predictions are. Specifically, the proposed ACD module spatially and temporally highlights the regions interacting with the extracted anchor in surgery video, which outputs action location and its class distribution based on anchor-context interactions. Considering the full distribution of action classes in videos, the CCD module adopts a denoising diffusion-based generative model conditioned on our ACD estimator to further reconstruct accurately the action predictions. Moreover, we utilize the stochastic nature of the diffusion model outputs to access model confidence for each prediction. Our method reports the state-of-the-art performance, with improvements of 4.0% mAP against baseline on the surgical video dataset.

Action detection Anchor-context Conditional diffusion Surgical video

共同第一 ; 通讯

英语

EI ; CPCI-S

General Program of National Natural Science Foundation of China["82272086","82102189"] ; Guangdong Basic and Applied Basic Research Foundation[2021A1515012195] ; Shenzhen Stable Support Plan Program["20220815111736001","20200925174052004"] ; Agency for Science, Technology and Research (A*STAR) Advanced Manufacturing and Engineering (AME) Programmatic Fund[A20H4b0141]

Computer Science ; Engineering

Computer Science, Artificial Intelligence ; Computer Science, Theory & Methods ; Engineering, Biomedical

WOS:001109638800019

Web of Science

1.School of Computer Science, University of Birmingham, Birmingham, United Kingdom
2.Research Institute of Trustworthy Autonomous Systems and Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China
3.Department of Mechanical Engineering, National University of Singapore, Singapore, Singapore
4.Third Medical Center of Chinese PLAGH, Beijing, China
5.Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR), Singapore, Singapore

Hao, Luoying,Hu, Yan,Lin, Wenjun,et al. ACT-Net: Anchor-Context Action Detection in Surgery Videos[C]. GEWERBESTRASSE 11, CHAM, CH-6330, SWITZERLAND:SPRINGER INTERNATIONAL PUBLISHING AG,2023.