An Autonomous Surgical Instrument Tracking Framework With a Binocular Camera for a Robotic Flexible Laparoscope

Li, Jian1; Huang, Yisen1; Zhang, Xue2; Xie, Ke1; Xian, Yitian1; Luo, Xiao1; Chiu, Philip Wai Yan1,3; Li, Zheng4,5

2023-07-01

10.1109/LRA.2023.3281934

IEEE ROBOTICS AND AUTOMATION LETTERS   影响因子和分区

2377-3766

In minimally invasive surgery (MIS), the field of view (FOV) plays a vital role. To enhance the stability of FOV and lighten the burden on surgeons, robot-assisted laparoscope systems have been developed and introduced into surgery. However, most of the existing automatic surgical tool tracking schemes with robotic laparoscopes either do not consider tracking of the depth direction or the remote center of motion (RCM) constraints. To tackle these problems, this article presents an autonomous three-dimensional surgical instrument tracking framework for a robotic laparoscope based on a binocular camera. A 3-degree-of-freedom visual tracking scheme is derived from the kinematics of the binocular robotic laparoscope. A programmable RCM constraint is introduced due to the MIS requirement. To precisely locate the surgical instruments in FOV, a surgical instrument tip segmentation dataset with 5471 images is built and a modified real-time instance segmentation network is trained based on the dataset. Then, a semi-global matching (SGM) algorithm is utilized with the binocular camera to estimate the depth in FOV. To meet the joint limit constraint and maintain the dexterity of the robotic system, a task-priority-based control method is adopted. The simulation and experiment results demonstrate the feasibility of the proposed framework for instrument tracking with the robotic laparoscope.

Instruments Laparoscopes Robots Three-dimensional displays Cameras Surgery Visualization Robotic laparoscopy binocular visual sensing Index Terms robot control visual servoing computer vision for surgical robotics

SCI ; EI

英语

其他

Hong Kong RGC General Research Fund["14203019","14202820","14214322"]

Robotics

Robotics

WOS:001005624000011

IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC

20232414219117

Binoculars ; Cameras ; Image segmentation ; Intelligent robots ; Laparoscopy ; Medical imaging ; Robot vision ; Robotic surgery ; Stereo image processing ; Surgical equipment ; Three dimensional computer graphics ; Three dimensional displays ; Transplantation (surgical)

Biomedical Engineering:461.1 ; Medicine and Pharmacology:461.6 ; Biomedical Equipment, General:462.1 ; Prosthetics:462.4 ; Computer Peripheral Equipment:722.2 ; Data Processing and Image Processing:723.2 ; Computer Applications:723.5 ; Robot Applications:731.6 ; Vision:741.2 ; Optical Devices and Systems:741.3 ; Photographic Equipment:742.2 ; Imaging Techniques:746

Web of Science

1.Chinese Univ Hong Kong, Dept Surg, Hong Kong, Peoples R China
2.Southern Univ Sci & Technol, Dept Elect & Elect Engn, Shen Zhen 518055, Peoples R China
3.Chinese Univ Hong Kong, Chow Yuk Ho Technol Ctr Innovat Med, Dept Surg, Hong Kong, Peoples R China
4.Chinese Univ Hong Kong, Li Ka Shing Inst Hlth Sci, Chow Yuk Ho Technol Ctr Innovat Med, Dept Surg, Hong Kong, Peoples R China
5.Chinese Univ Hong Kong, Multiscale Med Robot Ctr, Hong Kong, Peoples R China

Li, Jian,Huang, Yisen,Zhang, Xue,et al. An Autonomous Surgical Instrument Tracking Framework With a Binocular Camera for a Robotic Flexible Laparoscope[J]. IEEE ROBOTICS AND AUTOMATION LETTERS,2023,8(7):4291-4298.

Li, Jian.,Huang, Yisen.,Zhang, Xue.,Xie, Ke.,Xian, Yitian.,...&Li, Zheng.(2023).An Autonomous Surgical Instrument Tracking Framework With a Binocular Camera for a Robotic Flexible Laparoscope.IEEE ROBOTICS AND AUTOMATION LETTERS,8(7),4291-4298.

Li, Jian,et al."An Autonomous Surgical Instrument Tracking Framework With a Binocular Camera for a Robotic Flexible Laparoscope".IEEE ROBOTICS AND AUTOMATION LETTERS 8.7(2023):4291-4298.