Automatic Step Recognition with Video and Kinematic Data for Intelligent Operating Room and Beyond

Chng，Chin Boon1; Lin，Wenjun1; Hu，Yaxin1; Hu，Yan2; Liu，Jiang2; Chui，Chee Kong1

10.1145/3628797.3628999

2023-12-07

ACM International Conference Proceeding Series

599-606

With the continuous development of intelligent operating room systems, the segmentation and automatic recognition of surgical workflow have become challenging research fields. In recent years, an increasing number of models have been proposed to address this challenge, with deep learning becoming the mainstream approach. In this paper, we propose a multi-stage network for surgical step recognition by using surgical video and kinematic data. Firstly, a convolutional neural network (ResNet34) is used to extract visual features from video frames. Next, since surgical videos are a form of sequential data, a Temporal Convolutional Network (TCN) is employed as a temporal extractor to process temporal information between video frames for classification. Finally, a multi-stage TCN network, consisting of Encoder-Decoded TCN and Dilated TCN architectures, is used to refine the result. The proposed network is compared against a LSTM network from our prior work and is evaluated on a surgical dataset named MISAW in two modes - video data with and without kinematic data. Experimental results indicate that kinematic data is crucial for robot motion control in the operating rooms of the future. The technology will also find application in robotic labs for the development and optimization of chemical manufacturing processes.

Automation Intelligent Operating Room Multi-stage Model Step Recognition Surgical Robotics Temporal Convolutional Networks

其他

英语

EI

20235215293616

Classification (of information) ; Intelligent robots ; Kinematics ; Long short-term memory ; Operating rooms ; Surgery

Medicine and Pharmacology:461.6 ; Hospitals, Equipment and Supplies:462.2 ; Information Theory and Signal Processing:716.1 ; Robot Applications:731.6 ; Information Sources and Analysis:903.1 ; Mechanics:931.1

2-s2.0-85180550841

Scopus

1.National University of Singapore,Singapore,Singapore
2.Southern University of Science and Technology,Shenzhen,Guangdong,China

Chng，Chin Boon,Lin，Wenjun,Hu，Yaxin,et al. Automatic Step Recognition with Video and Kinematic Data for Intelligent Operating Room and Beyond[C],2023:599-606.