远程虚拟手术协同软组织交互方法研究

虚拟手术是一个新的交叉研究领域，它涵盖了医学、生物力学、计算机图形学和计算机视觉等多个学科。它的主要目的是通过计算机模拟和虚拟现实技术，将手术过程和手术器械与患者解剖结构进行三维可视化，以提高手术的准确性和安全性。然而，目前大多数的虚拟手术培训方案要求受训者和教学者在同一教学空间内，这较大程度地限制了培训的开展，并且现有方案主要针对单个受训者的场景，没有考虑多人协同的情况。远程虚拟手术有助于提高医疗效率、扩大医疗范围、提高医疗水平。

本文围绕虚拟手术软组织仿真交互方法的开展研究，针对现有虚拟手术仿真的四种主要方法进行了比较，提出了采用基于位置动力学方法进行仿真建模的工作，并实现了对缝线和软组织的模拟仿真。

本文使用传感器获取手术器械旋转角度、位移、张角等运动信息，并将真实场景中的运动信息映射到虚拟空间中，实现在虚拟场景中手术器械的跟踪定位和交互模拟。

本文使用共定位方法，实现手术培训的多人场景同步，在此基础上搭建了一个用于沉浸式远程虚拟手术协同训练的平台，并使用虚拟现实头显为外科医生提供沉浸式体验。

结果显示，本文实现了远程协同的多人虚拟手术培训，验证了该方法的有效性，并且有潜力成为医学元宇宙的基本单元。

[1] RICHTER G M, COLEMAN A L. Minimally invasive glaucoma surgery: current status and future prospects[J]. Clinical Ophthalmology (Auckland, NZ), 2016, 10: 189.
[2] MARESCAUX J, DIANA M. Next step in minimally invasive surgery: hybrid image-guided surgery[J]. Journal of pediatric surgery, 2015, 50(1): 30-36.
[3] SU H, MARIANI A, OVUR S E, et al. Toward teaching by demonstration for robot-assisted minimally invasive surgery[J]. IEEE Transactions on Automation Science and Engineering, 2021, 18(2): 484-494.
[4] ANTONIOU S A, ANTONIOU G A, ANTONIOU A I, et al. Past, present, and future of minimally invasive abdominal surgery[J]. JSLS: Journal of the Society of Laparoendoscopic Surgeons, 2015, 19(3).
[5] OVERTOOM E M, HOREMAN T, JANSEN F W, et al. Haptic feedback, force feedback, and force-sensing in simulation training for laparoscopy: A systematic overview[J]. Journal of surgical education, 2019, 76(1): 242-261.
[6] EVANS C H, SCHENARTS K D. Evolving educational techniques in surgical training[J]. Surgical Clinics, 2016, 96(1): 71-88.
[7] 郭园, 童倩倩, 郑宇铠, 等. 融合多元触觉和沉浸式视觉的可移植 VR 软件框架[J]. 系统仿真学报, 2020, 32(7): 1385.
[8] 吴一川, 孟欢欢, 黄启洋,等. 面向触觉力反馈的可穿戴柔性执行器研究现状[J]. 仪器仪表学报, 2021, 42(9):9.
[9] MOGLIA A, PIAZZA R, MOCELLIN D M, et al. Definition of proficiency level by a virtual simulator as a first step toward a curriculum on fundamental skills for endovascular aneurysm repair (EVAR)[J]. Journal of Surgical Education, 2020, 77(6): 1592-1597.
[10] BLUMSTEIN G, ZUKOTYNSKI B, CEVALLOS N, et al. Randomized trial of a virtual reality tool to teach surgical technique for tibial shaft fracture intramedullary nailing[J]. Journal of surgical education, 2020, 77(4): 969-977.
[11] ALZHRANI G, ALOTAIBI F, AZARNOUSH H, et al. Proficiency performance benchmarks for removal of simulated brain tumors using a virtual reality simulator NeuroTouch[J]. Journal of surgical education, 2015, 72(4): 685-696.
[12] WINKLER-SCHWARTZ A, YILMAZ R, MIRCHI N, et al. Machine learning identification of surgical and operative factors associated with surgical expertise in virtual reality simulation[J]. JAMA network open, 2019, 2(8): e198363-e198363.
[13] AYOUB A, PULIJALA Y. The application of virtual reality and augmented reality in Oral & Maxillofacial Surgery[J]. BMC Oral Health, 2019, 19(1): 1-8.
[14] GASQUES D, JOHNSON J G, SHARKEY T, et al. ARTEMIS: A collaborative mixed-reality system for immersive surgical telementoring[C]. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 2021: 1-14.
[15] HUANG T Y, LIN Y C, TSENG H Y, et al. Full percutaneous intraoperative neuromonitoring technique in remote thyroid surgery: porcine model feasibility study[J]. Head & Neck, 2021, 43(2): 505-513.
[16] ZHENG J, WANG Y, ZHANG J, et al. 5G ultra-remote robot-assisted laparoscopic surgery in China[J]. Surgical Endoscopy, 2020, 34: 5172-5180.
[17] 陈建江, 崔屏. 5G场景下的虚拟手术分析[J]. 科学咨询, 2020, 07.
[18] WISOTZKY E L, ROSENTHAL J C, EISERT P, et al. Interactive and multimodal-based augmented reality for remote assistance using a digital surgical microscope[C]. 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 2019: 1477-1484.
[19] MCGILLION M, OUELLETTE C, GOOD A, et al. Postoperative remote automated monitoring and virtual Hospital-to-Home Care system following cardiac and major vascular surgery: user testing study[J]. Journal of Medical Internet Research, 2020, 22(3): e15548.
[20] POSTOLACHE O, HEMANTH D J, ALEXANDRE R, et al. Remote monitoring of physical rehabilitation of stroke patients using IoT and virtual reality[J]. IEEE Journal on Selected Areas in Communications, 2020, 39(2): 562-573.
[21] HERRERA MONTANO I, PRESENCIO LAFUENTE E, BREÑOSA MARTÍNEZ J, et al. Systematic Review of Telemedicine and eHealth Systems Applied to Vascular Surgery[J]. Journal of Medical Systems, 2022, 46(12): 104.
[22] IRARRÁZAVAL M J, INZUNZA M, MUÑOZ R, et al. Telemedicine for postoperative follow-up, virtual surgical clinics during COVID-19 pandemic[J]. Surgical Endoscopy, 2021, 35: 6300-6306.
[23] ALTOBELLI D E, KIKINIS R, MULLIKEN J B, et al. Computer-assisted three-dimensional planning in craniofacial surgery[J]. Plastic and reconstructive surgery, 1993, 92(4): 576-585.
[24] QIAN K, BAI J, YANG X, et al. Virtual reality based laparoscopic surgery simulation[C]. Proceedings of the 21st ACM Symposium on Virtual Reality Software and Technology. 2015: 69-78.
[25] FAIRÉN M, MOYÉS J, INSA E. VR4Health: Personalized teaching and learning anatomy using VR[J]. Journal of medical systems, 2020, 44(5): 94.
[26] LOHRE R, BOIS A J, POLLOCK J W, et al. Effectiveness of immersive virtual reality on orthopedic surgical skills and knowledge acquisition among senior surgical residents: a randomized clinical trial[J]. JAMA Network Open, 2020, 3(12): e2031217-e2031217.
[27] TANG J, XU L, HE L, et al. Virtual laparoscopic training system based on VCH model[J]. Journal of medical systems, 2017, 41(4): 1-11.
[28] ALLAN M, OURSELIN S, THOMPSON S, et al. Toward detection and localization of instruments in minimally invasive surgery[J]. IEEE Transactions on Biomedical Engineering, 2012, 60(4): 1050-1058.
[29] PERRUSI P H S, BAKSIC P, COURTECUISSE H. Interactive Finite Element model of needle insertion and laceration[C]//Eurographics 2021-The 42nd Annual Conference of the European Association for Computer Graphics. 2021.
[30] BAKSIC P, COURTECUISSE H, DURIEZ C, et al. Robotic needle insertion in moving soft tissues using constraint-based inverse Finite Element simulation[C]//2020 IEEE international conference on robotics and automation (ICRA). IEEE, 2020: 2407-2413.
[31] 彭柳, 司伟鑫, 郑睿,等. 高保真电视胸腔镜肺叶切除术模拟[J]. 系统仿真学报, 2020, 32(5):11.
[32] 戴莎, 司伟鑫, 钱银玲,等. 虚拟显微白内障手术系统人机交互接口设计[J]. 图学学报, 2019, 40(3):9.
[33] LACY A M, BRAVO R, OTERO‐PINEIRO A M, et al. 5G‐assisted telementored surgery[J]. British Journal of Surgery, 2019, 106(12): 1576-1579.
[34] HUANG T, LI R, LI Y, et al. Augmented reality-based autostereoscopic surgical visualization system for telesurgery[J]. International Journal of Computer Assisted Radiology and Surgery, 2021, 16(11): 1985-1997.
[35] SADEGHI A H, WAHADAT A R, DERECI A, et al. Remote multidisciplinary heart team meetings in immersive virtual reality: a first experience during the COVID-19 pandemic[J]. BMJ innovations, 2021, 7(2).
[36] YOON J W, WELCH R L, ALAMIN T, et al. Remote virtual spinal evaluation in the era of COVID-19[J]. International Journal of Spine Surgery, 2020, 14(3): 433-440.
[37] TEA S, PANUWATWANICH K, RUTHANKOON R, et al. Multiuser immersive virtual reality application for real-time remote collaboration to enhance design review process in the social distancing era[J]. Journal of Engineering, Design and Technology, 2022, 20(1): 281-298.
[38] LU Y, XU Y, ZHU X. Designing and Implementing VR2E2C, a Virtual Reality Remote Education for Experimental Chemistry System[J]. Journal of Chemical Education, 2021, 2720-2725.
[39] MAKHNI M C, RIEW G J, SUMATHIPALA M G. Telemedicine in orthopaedic surgery: challenges and opportunities[J]. JBJS, 2020, 102(13): 1109-1115.
[40] VYAS R M, SAYADI L R, BENDIT D, et al. Using virtual augmented reality to remotely proctor overseas surgical outreach: building long-term international capacity and sustainability[J]. Plastic and Reconstructive Surgery, 2020, 146(5): 622e-629e.
[41] FEREYDOONI N , WALKER B N . Virtual Reality as a Remote Workspace Platform: Opportunities and Challenges[C]. Microsoft New Future of Work Virtual Symposium. 2020.
[42] ALMOUSA O, ZHANG R, DIMMA M, et al. Virtual reality technology and remote digital application for tele-simulation and global medical education: an innovative hybrid system for clinical training[J]. Simulation & Gaming, 2021, 52(5): 614-634.
[43] LIU P, LI C, XIAO C, et al. A wearable augmented reality navigation system for surgical telementoring based on Microsoft HoloLens[J]. Annals of biomedical engineering, 2021, 49: 287-298.
[44] AMINOFF H, MEIJER S, ARNELO U, et al. Telemedicine for remote surgical guidance in endoscopic retrograde cholangiopancreatography: mixed methods study of practitioner attitudes[J]. JMIR formative research, 2021, 5(1): e20692.
[45] NICKEL F, CIZMIC A, CHAND M. Telestration and augmented reality in minimally invasive surgery: an invaluable tool in the age of COVID-19 for remote proctoring and telementoring[J]. JAMA surgery, 2022, 157(2): 169-170.
[46] POSTOLACHE O, HEMANTH D J, ALEXANDRE R, et al. Remote monitoring of physical rehabilitation of stroke patients using IoT and virtual reality[J]. IEEE Journal on Selected Areas in Communications, 2020, 39(2): 562-573.
[47] SHABIR D, ABDURAHIMAN N, PADHAN J, et al. Towards development of a tele‐mentoring framework for minimally invasive surgeries[J]. The International Journal of Medical Robotics and Computer Assisted Surgery, 2021, 17(5): e2305.
[48] YOUNUS O, GILLAM M, CAUCHI P, et al. Real world outcomes of a virtual ocular oncology service in Scotland[J]. Eye, 2022: 1-3.
[49] 赵刚, 王能才, 韦哲,等. 基于5G的移动通讯技术在远程医疗中的应用[J]. 中国医学装备, 2020, 17(10):4.
[50] 岳梅. 虚拟现实技术在远程医学教学中的应用场景[J]. 中国中医药现代远程教育, 2020, 18(21):3.
[51] 袁帅, 陈斌, 易超, 等. 虚拟地理环境中沉浸式多人协同交互技术研究及实现[J]. 地球信息科学学报, 2018, 20(8): 1055-1063.
[52] 熊山, 郭建龙, 祁彦威, 等. 基于多人协同虚拟现实的主变应急培训应用研究[J]. 计算技术与自动化, 2021 (4): 145-150.
[53] MACKLIN M, MÜLLER M, CHENTANEZ N, et al. Unified particle physics for real-time applications[J]. ACM Transactions on Graphics (TOG), 2014, 33(4): 1-12.
[54] MACKLIN M, MÜLLER M, CHENTANEZ N. XPBD: position-based simulation of compliant constrained dynamics[C]. Proceedings of the 9th International Conference on Motion in Games. 2016: 49-54.
[55] MACKLIN M, STOREY K, LU M, et al. Small steps in physics simulation[C]. Proceedings of the 18th Annual ACM SIGGRAPH/Eurographics Symposium on Computer Animation. 2019: 1-7.
[56] EITZ M, LIXU G. Hierarchical spatial hashing for real-time collision detection[C]. IEEE International Conference on Shape Modeling and Applications 2007 (SMI'07). IEEE, 2007: 61-70.
[57] MACKLIN M, ERLEBEN K, MULLER M, et al. Local optimization for robust signed distance field collision[J]. Proceedings of the ACM on Computer Graphics and Interactive Techniques, 2020, 3(1): 1-17.
[58] KUBIAK B, PIETRONI N, GANOVELLI F, et al. A robust method for real-time thread simulation[C]. Proceedings of the 2007 ACM symposium on Virtual reality software and technology. 2007: 85-88.
[59] KUGELSTADT T, SCHÖMER E. Position and orientation based Cosserat rods[C]. Symposium on Computer Animation. 2016: 169-178.
[60] XU L, LIU Q. Real-time inextensible surgical thread simulation[J]. International Journal of Computer Assisted Radiology and Surgery, 2018, 13: 1019-1035.
[61] MÜLLER M, HEIDELBERGER B, TESCHNER M, et al. Meshless deformations based on shape matching[J]. ACM transactions on graphics (TOG), 2005, 24(3): 471-478.
[62] MÜLLER M, CHENTANEZ N. Solid simulation with oriented particles[M]. ACM SIGGRAPH 2011 papers. 2011: 1-10.
[63] BRIDSON R, FEDKIW R, ANDERSON J. Robust treatment of collisions, contact and friction for cloth animation[C]. Proceedings of the 29th annual conference on Computer graphics and interactive techniques. 2002: 594-603.
[64] KELAGER M, NIEBE S, ERLEBEN K. A Triangle Bending Constraint Model for Position-Based Dynamics[J]. VRIPHYS, 2010, 10: 31-37.
[65] KIM T Y, CHENTANEZ N, MÜLLER-FISCHER M. Long range attachments-a method to simulate inextensible clothing in computer games[C]. Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation. 2012: 305-310.
[66] SOUZA M S, WANGENHEIM A, COMUNELLO E. Fast simulation of cloth tearing[J]. SBC Journal on Interactive Systems, 2014, 5(1): 44-48.
[67] MÜLLER M, BENDER J, CHENTANEZ N, et al. A robust method to extract the rotational part of deformations[C]. Proceedings of the 9th International Conference on Motion in Games. 2016: 55-60.
[68] WU H, YE F, GAO Y, et al. Real-Time Laparoscopic Cholecystectomy Simulation Using a Particle-Based Physical System[J]. Complex System Modeling and Simulation, 2022, 2(2): 186-196.
[69] ALLAN M, OURSELIN S, THOMPSON S, et al. Toward detection and localization of instruments in minimally invasive surgery[J]. IEEE Transactions on Biomedical Engineering, 2012, 60(4): 1050-1058.
[70] 冒苏敏, 周乐, 张金宇,等. 信息物理融合系统应用前景研究[J]. 中国新通信, 2016, 18(21):1.
[71] QIAN K, BAI J, YANG X, et al. Essential techniques for laparoscopic surgery simulation[J]. Computer Animation and Virtual Worlds, 2017, 28(2): e1724.
[72] TURCHETTI G, PALLA I, PIEROTTI F, et al. Economic evaluation of da Vinci-assisted robotic surgery: a systematic review[J]. Surgical endoscopy, 2012, 26(3): 598-606.
[73] ARIKATLA V S, SANKARANARAYANAN G, AHN W, et al. Face and construct validation of a virtual peg transfer simulator[J]. Surgical endoscopy, 2013, 27(5): 1721-1729.
[74] MENDIZABAL A, TAGLIABUE E, HOELLINGER T, et al. Data-driven simulation for augmented surgery[J]. Developments and Novel Approaches in Biomechanics and Metamaterials, 2020: 71-96.
[75] HARRINGTON C M, DICKER P, TRAYNOR O, et al. Visuospatial abilities and fine motor experiences influence acquisition and maintenance of fundamentals of laparoscopic surgery (FLS) task performance[J]. Surgical endoscopy, 2018, 32: 4639-4648.
[76] SU H, YANG C, FERRIGNO G, et al. Improved human–robot collaborative control of redundant robot for teleoperated minimally invasive surgery[J]. IEEE Robotics and Automation Letters, 2019, 4(2): 1447-1453.
[77] LAAKI H, MICHE Y, TAMMI K. Prototyping a digital twin for real time remote control over mobile networks: Application of remote surgery[J]. IEEE Access, 2019, 7: 20325-20336.
[78] RUSSELL J O, RAZAVI C R, GARSTKA M E, et al. Remote-access thyroidectomy: a multi-institutional North American experience with transaxillary, robotic facelift, and transoral endoscopic vestibular approaches[J]. Journal of the American College of Surgeons, 2019, 228(4): 516-522.
[79] TERRIS D J, SINGER M C. Robotic facelift thyroidectomy: facilitating remote access surgery[J]. Head & neck, 2012, 34(5): 746-747.
[80] CASTILLEJO J A P. Telemedicina, una herramienta también para el médico de familia[J]. Atención primaria, 2013, 45(3): 129.
[81] HUANG E Y, KNIGHT S, GUETTER C R, et al. Telemedicine and telementoring in the surgical specialties: a narrative review[J]. The American Journal of Surgery, 2019, 218(4): 760-766.
[82] PORTNOY J, WALLER M, ELLIOTT T. Telemedicine in the era of COVID-19[J]. The Journal of Allergy and Clinical Immunology: In Practice, 2020, 8(5): 1489-1491.
[83] 李晋芳, 陈基荣, 李二芳,等. 远程协同手术指导平台的研究[J]. 实验室研究与探索, 2019, 38(6):7.
[84] ODOT A, HAFERSSAS R, COTIN S. DeepPhysics: a physics aware deep learning framework for real-time simulation[J]. arXiv preprint arXiv:2109.09491, 2021.
[85] PFAFF T, FORTUNATO M, SANCHEZ-GONZALEZ A, et al. Learning Mesh-Based Simulation with Graph Networks[J]. arXiv preprint arXiv:2010.03409, 2020.
[86] SUZUKI R, KARIM A, XIA T, et al. Augmented reality and robotics: A survey and taxonomy for ar-enhanced human-robot interaction and robotic interfaces[C]. Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 2022: 1-33.
[87] WEN R, YANG L, CHUI C, et al. Intraoperative visual guidance and control interface for augmented reality robotic surgery[C]. IEEE International Conference on Control & Automation. IEEE, 2010.
[88] OVERTOOM E M, HOREMAN T, JANSEN F W, et al. Haptic feedback, force feedback, and force-sensing in simulation training for laparoscopy: A systematic overview[J]. Journal of surgical education, 2019, 76(1): 242-261.