3D Reconstruction of Colon Structures and Textures from Colonoscopic Videos

Colonoscopy is considered the most effective method for detecting and removing precan-cerous polyps in the human colon. This procedure uses an endoscope to examine the internal surface of the entire colon. However, during a standard colonoscopy, it can be challenging for the endoscopist to ensure that the entire colon internal surface is inspected from the colon screening video, which can result in missed polyps and adenomas in unin-spected regions. If a 3D map of the colon internal surface with detailed textures can be reconstructed during the colonoscopy procedure, the following two main potential benefits can be achieved: i) uninspected regions can be shown on this map and the endoscopist can navigate the endoscope to these missing regions to ensure more colon surfaces are inspected; ii) the detailed textures on the reconstructed map can help the endoscopist to inspect abnormalities offline.

In this dissertation, we present three works for reconstructing 3D colon maps from colono-scopic videos. Meanwhile, we introduce a colonoscopy simulator developed in Unity that can simulate the procedures of colonoscopy, different levels of colonic surface deformation, and generate synthetic colonoscopy datasets in different scenarios for the development and validation of colon reconstruction algorithms. Furthermore, to foster research in this field, the colonoscopy simulator and source code are made publicly available.

The first work presents a framework for 3D reconstruction of the colonic surface using stereo colonoscopic images. The input comprises a sequence of stereo colonoscopic images and a corresponding colon mesh model, which is segmented from pre-operative CT scans. The final output is the reconstructed and texturized 3D colon maps. The primary contribution of this work is fourfold: (1) Developing a visual odometry for endoscopic camera pose initialization; (2) Using the pre-operative CT-segmented colon model as a global colon map reference to increase the stability and accuracy of the endoscopic camera pose estimation; (3) Developing a joint photometric and geometric constrained scan-to-model registration algorithm for matching 3D scans (point cloud with RGB information and reconstructed from stereo images) to the pre-operative CT-segmented colon model, which can address the inconsistency of the texture matching problem; (4) Developing a barycentric-based texture rendering module for mapping textures from colonoscopic images to the reconstructed colonic surface. Simulation experimental results demonstrate the feasibility and good performance of the proposed 3D colonic surface reconstruction method in terms of accuracy and robustness.

In a clinical setting, the majority of colonoscopes used for colonoscopy procedures are equipped with a monocular camera. Meanwhile, the 3D reconstruction of colonic surface faces the problem of colon deformation. To improve the practicability of the first proposed framework, in the second work, we present a framework that can recover the 3D shape of deformable colon structures with textures from monocular colonoscopic images and a corresponding pre-operative CT-segmented colon mesh model. The novelty of the second work is threefold: (1) Using deep learning techniques to estimate dense depth for monocular colonoscopic images; (2) Developing a non-rigid registration method to address the problem of colon deformation; and (3) Developing the entire framework for the 3D reconstruction of deformable colonic surfaces with high accuracy. Validation by simulation and in-vivo experiments is conducted, and the results demonstrate the practicality of the non-rigid 3D colon reconstruction framework.

The third work is significantly differs from the previous two works, which require pair-wise photometric correspondences and dense geometric correspondences, posing a great challenge for low-textured colonoscopic images. In the third work, we formulate the textured colon reconstruction problem as a bundle adjustment (BA) problem where all the camera poses and the intensities of mesh model vertices are jointly optimized by maximizing the photometric consistency between the pre-operative CT-segmented colon mesh model and multiple views of colonoscopic images. Then, the optimized camera poses are used to render the colon map with textures from colonoscopic images. The novelty of this work is threefold: (1) Formulating simultaneous camera pose estimation as a direct BA problem, where the pre-operative model intensities and all camera poses are jointly optimized, which differs from traditional BA; (2) Directly using intensity information avoids the feature extraction and matching between 2D images in traditional BA, making the proposed method applicable to images lacking salient features, such as colonoscopic images; (3) We prove that when solving the proposed BA problem using the Gauss-Newton (GN) algorithm, the pose estimation result in each iteration of GN is independent of the model intensities in the previous iteration step, thus we propose the camera-only BA algorithm which is equivalent to the proposed direct BA algorithm but with less computational cost. The practicality and accuracy of the proposed direct camera-only BA method are validated using simulation, phantom, and in-vivo datasets.verall, the three frameworks proposed in this thesis represent a notable advancement in the field of 3D colonic surface reconstruction, using colonoscopic images and a pre-operative CT-segmented colon mesh model. These frameworks undergo validation through rigorous testing with simulation, phantom, and in-vivo datasets, demonstrating their feasibility, accuracy, and practicality. The clinical applications of these frameworks have the potential to enhance the accuracy and efficiency of colonic surface 3D reconstruction, thereby benefiting diagnosis, treatment planning, and surgical navigation in colonoscopy procedures.

[1] Colonoscopy, the worlds of david darling. https://www.daviddarling.info/encyclopedia/C/colonoscopy.html, 2016. Accessed: 2016.
[2] AM Leufkens, MGH Van Oijen, FP Vleggaar, and PD Siersema. Factors influencingthe miss rate of polyps in a back-to-back colonoscopy study. Endoscopy, 44(05): 470–475, 2012.
[3] Jeroen C Van Rijn, Johannes B Reitsma, Jaap Stoker, Patrick M Bossuyt, Sander JVan Deventer, and Evelien Dekker. Polyp miss rate determined by tandem colonoscopy: a systematic review. Official journal of the American College of Gastroenterology— ACG, 101(2):343–350, 2006.
[4] Chantal MC le Clercq, Mari¨elle WE Bouwens, Eveline JA Rondagh, C Minke Bakker,Eric TP Keulen, Rogier J de Ridder, Bjorn Winkens, Ad AM Masclee, and Silvia Sanduleanu. Postcolonoscopy colorectal cancers are preventable: a population-based study. Gut, 63(6):957–963, 2014.
[5] Jerome D Waye. Difficult colonoscopy. Gastroenterology & Hepatology, 9(10):676, 2013.
[6] Hongbin Zhu, Matthew Barish, Perry Pickhardt, and Zhengrong Liang. Haustralfold segmentation with curvature-guided level set evolution. IEEE Transactions onBiomedical Engineering, 60(2):321–331, 2012.
[7] Andrew Guinigundo. Is the virtual colonoscopy a replacement for optical colonoscopy? In Seminars in oncology nursing, volume 34, pages 132–136. Elsevier, 2018.
[8] Donald W Marquardt. An algorithm for least-squares estimation of nonlinear param-eters. Journal of the society for Industrial and Applied Mathematics, 11(2):431–441, 1963.
[9] Robert W Sumner, Johannes Schmid, and Mark Pauly. Embedded deformation forshape manipulation. ACM Transactions on Graphics (TOG), 26(3):80, 2007.
[10] Olek C Zienkiewicz, Robert Leroy Taylor, and Jian Z Zhu. The finite element method:its basis and fundamentals. Elsevier, 2005.
[11] Fernando De Goes and Doug L James. Regularized kelvinlets: sculpting brushesbased on fundamental solutions of elasticity. ACM Transactions on Graphics (TOG),36(4):1–11, 2017.
[12] Fred L. Bookstein. Principal warps: Thin-plate splines and the decomposition ofdeformations. IEEE Transactions on pattern analysis and machine intelligence, 11(6):567–585, 1989.
[13] Marco Riva, Patrick Hiepe, Mona Frommert, Ignazio Divenuto, Lorenzo G Gay, Tom-maso Sciortino, Marco Conti Nibali, Marco Rossi, Federico Pessina, and LorenzoBello. Intraoperative computed tomography and finite element modelling for multi-modal image fusion in brain surgery. Operative Neurosurgery, 18(5):531–541, 2020.
[14] Sarah Frisken, Ma Luo, Parikshit Juvekar, Adomas Bunevicius, Ines Machado,Prashin Unadkat, Melina M Bertotti, Matt Toews, William M Wells, Michael I Miga,et al. A comparison of thin-plate spline deformation and finite element modeling tocompensate for brain shift during tumor resection. International journal of computerassisted radiology and surgery, 15:75–85, 2020.
[15] Daniel E Hurtado, Nicol´as Villarroel, Jaime Retamal, Guillermo Bugedo, and Ale-jandro Bruhn. Improving the accuracy of registration-based biomechanical analysis:a finite element approach to lung regional strain quantification. IEEE Transactionson Medical Imaging, 35(2):580–588, 2015.
[16] Hongjian Shi. Finite element modeling of soft tissue deformation. University ofLouisville, 2007.
[17] Zhiyu Qiu, Huihui Tang, and Dongsheng Tian. Non-rigid medical image registrationbased on the thin-plate spline algorithm. In 2009 WRI World Congress on ComputerScience and Information Engineering, volume 2, pages 522–527. IEEE, 2009.
[18] Jian Zhao and Hui Zhang. Thin-plate spline motion model for image animation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni-tion, pages 3657–3666, 2022.
[19] Mohammad Ali Armin, Nick Barnes, Jose Alvarez, Hongdong Li, Florian Grimpen,and Olivier Salvado. Learning camera pose from optical colonoscopy frames throughdeep convolutional neural network (cnn). In Computer Assisted and Robotic En-doscopy and Clinical Image-Based Procedures: 4th International Workshop, CARE2017, and 6th International Workshop, CLIP 2017, Held in Conjunction with MIC-CAI 2017, Qu´ebec City, QC, Canada, September 14, 2017, Proceedings 4, pages 50–59. Springer, 2017.
[20] Anita Rau, PJ Eddie Edwards, Omer F Ahmad, Paul Riordan, Mirek Janatka, Lau-rence B Lovat, and Danail Stoyanov. Implicit domain adaptation with conditionalgenerative adversarial networks for depth prediction in endoscopy. International jour-nal of computer assisted radiology and surgery, 14:1167–1176, 2019.
[21] Gwangbin Bae, Ignas Budvytis, Chung-Kwong Yeung, and Roberto Cipolla. Deepmulti-view stereo for dense 3d reconstruction from monocular endoscopic video. InInternational Conference on Medical Image Computing and Computer-Assisted Inter-vention, pages 774–783. Springer, 2020.
[22] Daniel Freedman, Yochai Blau, Liran Katzir, Amit Aides, Ilan Shimshoni, DannyVeikherman, Tomer Golany, Ariel Gordon, Greg Corrado, Yossi Matias, et al. Detecting deficient coverage in colonoscopies. IEEE Transactions on Medical Imaging, 39(11):3451–3462, 2020.
[23] Mitchell J Fulton, J Micah Prendergast, Emily R DiTommaso, and Mark E Rentschler. Comparing visual odometry systems in actively deforming simulated colon environments. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4988–4995. IEEE, 2020.
[24] Kutsev Bengisu Ozyoruk, Guliz Irem Gokceler, Taylor L Bobrow, Gulfize Coskun, Ka-gan Incetan, Yasin Almalioglu, Faisal Mahmood, Eva Curto, Luis Perdigoto, MarinaOliveira, et al. Endoslam dataset and an unsupervised monocular visual odometryand depth estimation approach for endoscopic videos. Medical image analysis, 71:102058, 2021.
[25] Ruibin Ma, Rui Wang, Yubo Zhang, Stephen Pizer, Sarah K McGill, Julian Rosen-man, and Jan-Michael Frahm. Rnnslam: Reconstructing the 3d colon to visualizemissing regions during a colonoscopy. Medical image analysis, 72:102100, 2021.
[26] Taylor L Bobrow, Mayank Golhar, Rohan Vijayan, Venkata S Akshintala, Juan RGarcia, and Nicholas J Durr. Colonoscopy 3d video dataset with paired depth from2d-3d registration. arXiv preprint arXiv:2206.08903, 2022.
[27] Faisal Mahmood, Richard Chen, and Nicholas J Durr. Unsupervised reverse domainadaptation for synthetic medical images via adversarial training. IEEE transactionson medical imaging, 37(12):2572–2581, 2018.
[28] Berthold KP Horn. A method for obtaining the shape of a smooth opaque objectfrom one view. PhD thesis, Massachusetts Institute of Technology, Cambridge, 1970,1970.
[29] Alexandros Karargyris and Nikolaos Bourbakis. Three-dimensional reconstruction ofthe digestive wall in capsule endoscopy videos using elastic video interpolation. IEEEtransactions on Medical Imaging, 30(4):957–971, 2010.
[30] Dan Koppel, Chao-I Chen, Yuan-Fang Wang, Hua Lee, Jia Gu, Allen Poirson, andRolf Wolters. Toward automated model building from video in computer-assisteddiagnoses in colonoscopy. In Medical Imaging 2007: Visualization and Image-GuidedProcedures, volume 6509, page 65091L. International Society for Optics and Photon-ics, 2007.
[31] Chao-I Chen, Dusty Sargent, and Yuan-Fang Wang. Modeling tumor/polyp/lesionstructure in 3d for computer-aided diagnosis in colonoscopy. In Medical Imaging 2010:Visualization, Image-Guided Procedures, and Modeling, volume 7625, page 76252F.International Society for Optics and Photonics, 2010.
[32] Jin Zhou, Ananya Das, Feng Li, and Baoxin Li. Circular generalized cylinder fittingfor 3d reconstruction in endoscopic imaging based on mrf. In IEEE Computer Soci-ety Conference on Computer Vision and Pattern Recognition Workshops, pages 1–8.IEEE, 2008.
[33] DongHo Hong, Wallapak Tavanapong, Johnny Wong, JungHwan Oh, and Piet CDe Groen. 3d reconstruction of virtual colon structures from colonoscopy images.Computerized Medical Imaging and Graphics, 38(1):22–33, 2014.
[34] Mohammad Ali Armin, Girija Chetty, Hans De Visser, Cedric Dumas, FlorianGrimpen, and Olivier Salvado. Automated visibility map of the internal colon surfacefrom colonoscopy video. International Journal of Computer Assisted Radiology andSurgery, 11(9):1599–1610, 2016.
[35] Faisal Mahmood and Nicholas J Durr. Deep learning and conditional random fields-based depth estimation and topographical reconstruction from conventional endoscopy. Medical image analysis, 48:230–243, 2018.
[36] Xingtong Liu, Ayushi Sinha, Masaru Ishii, Gregory D Hager, Austin Reiter, Russell HTaylor, and Mathias Unberath. Dense depth estimation in monocular endoscopy withself-supervised learning methods. IEEE transactions on medical imaging, 39(5):1438–1447, 2019.
[37] Shawn Mathew, Saad Nadeem, Sruti Kumari, and Arie Kaufman. Augmentingcolonoscopy using extended and directional cyclegan for lossy image translation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4696–4705, 2020.
[38] J Kowalczuk, A Meyer, J Carlson, ET Psota, S Buettner, LC P´erez, SM Farritor, andD Oleynikov. Real-time three-dimensional soft tissue reconstruction for laparoscopicsurgery. Surgical Endoscopy, 26(12):3413–3417, 2012.
[39] Nicholas J Durr, Germ´an Gonz´alez, and Vicente Parot. 3d imaging techniques forimproved colonoscopy, 2014.
[40] Alberto Arezzo, Nereo Vettoretto, Nader K Francis, Marco Augusto Bonino, Nathan JCurtis, Daniele Amparore, Simone Arolfo, Manuel Barberio, Luigi Boni, Ronit Brodie, et al. The use of 3d laparoscopic imaging systems in surgery: Eaes consensus development conference 2018. Surgical endoscopy, 33(10):3251–3274, 2019.
[41] Tobias Bergen and Thomas Wittenberg. Stitching and surface reconstruction fromendoscopic image sequences: a review of applications and methods. IEEE journal ofbiomedical and health informatics, 20(1):304–321, 2014.
[42] Raul Mur-Artal and Juan D Tard´os. Orb-slam2: An open-source slam system formonocular, stereo, and rgb-d cameras. IEEE Transactions on Robotics, 33(5):1255–1262, 2017.
[43] Georg Klein and David Murray. Parallel tracking and mapping for smallar workspacs. In 2007 6th IEEE and ACM international symposium on mixed and augmented reality, pages 225–234. IEEE, 2007.
[44] Jakob Engel, Vladlen Koltun, and Daniel Cremers. Direct sparse odometry. IEEEtransactions on pattern analysis and machine intelligence, 40(3):611–625, 2017.
[45] Jakob Engel, Thomas Sch¨ops, and Daniel Cremers. Lsd-slam: Large-scale directmonocular slam. In European conference on computer vision, pages 834–849. Springer,2014.
[46] Christian Forster, Zichao Zhang, Michael Gassner, Manuel Werlberger, and DavideScaramuzza. Svo: Semidirect visual odometry for monocular and multicamera sys-tems. IEEE Transactions on Robotics, 33(2):249–265, 2016.
[47] Rui Wang, Martin Schworer, and Daniel Cremers. Stereo dso: Large-scale directsparse visual odometry with stereo cameras. In Proceedings of the IEEE InternationalConference on Computer Vision, pages 3903–3911, 2017.
[48] Thomas Whelan, Renato F Salas-Moreno, Ben Glocker, Andrew J Davison, and Ste-fan Leutenegger. Elasticfusion: Real-time dense slam and light source estimation.The International Journal of Robotics Research, 35(14):1697–1716, 2016.
[49] M Kaess, M Fallon, H Johannsson, and JJ Leonard. Kintinuous: Spatially extendedkinectfusion. In Proceedings of the RSS Workshop on RGB-D: Advanced Reasoningwith Depth Cameras, Sydney, Australia, pages 9–10, 2012.
[50] Richard A Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, DavidKim, Andrew J Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon. Kinectfusion: Real-time dense surface mapping and tracking. In 2011 10th IEEE International Symposium on Mixed and Augmented Reality, pages 127–136. IEEE, 2011.
[51] Richard J Chen, Taylor L Bobrow, Thomas Athey, Faisal Mahmood, and Nicholas JDurr. Slam endoscopy enhanced by adversarial depth prediction. arXiv preprintarXiv:1907.00283, 2019.
[52] Ruibin Ma, Rui Wang, Stephen Pizer, Julian Rosenman, Sarah K McGill, and Jan-Michael Frahm. Real-time 3d reconstruction of colonoscopic surfaces for determin-ing missing regions. In International Conference on Medical Image Computing andComputer-Assisted Intervention, pages 573–582. Springer, 2019.
[53] Johannes Lutz Sch¨onberger and Jan-Michael Frahm. Structure-from-motion revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[54] Ama¨el Delaunoy and Marc Pollefeys. Photometric bundle adjustment for dense multiview 3d modeling. In Proceedings of the IEEE Conference on Computer Vision andPattern Recognition, pages 1486–1493, 2014.
[55] Hatem Alismail, Brett Browning, and Simon Lucey. Photometric bundle adjustmentfor vision-based slam. In Computer Vision–ACCV 2016: 13th Asian Conference on Computer Vision, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part IV 13, pages 324–341. Springer, 2017.
[56] Zhehua Mao, Liang Zhao, Shoudong Huang, Yiting Fan, and Alex PW Lee. Dsr:Direct simultaneous registration for multiple 3d images. In Medical Image Computingand Computer Assisted Intervention–MICCAI 2022: 25th International Conference,Singapore, September 18–22, 2022, Proceedings, Part VI, pages 98–107. Springer,2022.
[57] John K Haas. A history of the unity game engine. Diss. Worcester PolytechnicInstitute, 483:484, 2014.
[58] Ron Kikinis, Steve D Pieper, and Kirby G Vosburgh. 3d slicer: a platform for subject specific image analysis, visualization, and clinical support. In Intraoperative imagingand image-guided therapy, pages 277–289. Springer, 2013.
[59] Eric Keller. Introducing ZBrush. John Wiley & Sons, 2011.
[60] Ingrid Hoelzl and R´emi Marie. Softimage: Towards a new theory of the digital image. Intellect Books, 2015.
[61] Wikipedia contributors. Adobe photoshop — Wikipedia, the free encyclopedia, 2023. URL https://en.wikipedia.org/w/index.php?title=Adobe_Photoshop&oldid=1146911454. [Online; accessed 31-March-2023].
[62] Ruibin Ma, Qingyu Zhao, Rui Wang, James Damon, Julian Rosenman, and StephenPizer. Deforming generalized cylinders without self-intersection by means of a para-metric center curve. Computational Visual Media, 4(4):305–321, 2018.
[63] Heiko Hirschmuller. Accurate and efficient stereo processing by semi-global matching and mutual information. In 2005 IEEE Computer Society Conference on ComputerVision and Pattern Recognition, volume 2, pages 807–814. IEEE, 2005.
[64] Friedrich Fraundorfer and Davide Scaramuzza. Visual odometry: Part i: The first30 years and fundamentals. IEEE Robotics and Automation Magazine, 18(4):80–92, 2011.
[65] Xiao-Shan Gao, Xiao-Rong Hou, Jianliang Tang, and Hang-Fei Cheng. Completesolution classification for the perspective-three-point problem. IEEE transactions onpattern analysis and machine intelligence, 25(8):930–943, 2003.
[66] Philip HS Torr and Andrew Zisserman. Mlesac: A new robust estimator with appli-cation to estimating image geometry. Computer vision and image understanding, 78(1):138–156, 2000.
[67] Szymon Rusinkiewicz and Marc Levoy. Efficient variants of the icp algorithm. InProceedings Third International Conference on 3D Digital Imaging and Modeling,pages 145–152. IEEE, 2001.
[68] Ofir Weber, Mirela Ben-Chen, Craig Gotsman, and Kai Hormann. A complex viewof barycentric mappings. In Computer Graphics Forum, volume 30, pages 1533–1542.Wiley Online Library, 2011.
[69] Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe,Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison,et al. Kinectfusion: real-time 3d reconstruction and interaction using a moving depthcamera. In Proceedings of the 24th annual ACM symposium on User interface softwareand technology, pages 559–568, 2011.
[70] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of theIEEE international conference on computer vision, pages 2223–2232, 2017.
[71] Ibraheem Alhashim and Peter Wonka. High quality monocular depth estimation viatransfer learning. arXiv preprint arXiv:1812.11941, 2018.
[72] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image qualityassessment: from error visibility to structural similarity. IEEE transactions on imageprocessing, 13(4):600–612, 2004.
[73] S Katz. Direct visibility of point. sets. ACM Trans Graphics, 26(3):24–1, 2007.
[74] Pierre Biasutti, Aur´elie Bugeau, Jean-Fran¸cois Aujol, and Mathieu Br´edif. Visibilityestimation in point clouds with variable density. In VISIGRAPP (4: VISAPP), pages27–35, 2019.
[75] Bill Triggs, Philip F McLauchlan, Richard I Hartley, and AndrewWFitzgibbon. Bun-dle adjustment—a modern synthesis. In International workshop on vision algorithms,pages 298–372. Springer, 1999.
[76] Lukas Polok, Viorela Ila, Marek Solony, Pavel Smrz, and Pavel Zemcik. Incremen-tal block cholesky factorization for nonlinear least squares in robotics. In Robotics:Science and Systems, pages 328–336, 2013.
[77] Fuzhen Zhang. The Schur complement and its applications, volume 4. Springer Science & Business Media, 2006.
[78] Yubo Zhang, Shuxian Wang, Ruibin Ma, Sarah K McGill, Julian G Rosenman, andStephen M Pizer. Lighting enhancement aids reconstruction of colonoscopic surfaces.In Information Processing in Medical Imaging: 27th International Conference, IPMI2021, Virtual Event, June 28–June 30, 2021, Proceedings 27, pages 559–570. Springer,2021.