融合激光雷达与相机的三维重建

近几年来，随着游戏建模行业的发展，以及虚拟现实、增强现实等应用的崛 起，三维重建技术逐渐成为当前研究的热点，传感器硬件的快速更迭也让三维重 建技术的高速发展成为了现实。在现实应用场景中，常常涉及到大规模场景的三 维重建，而大规模场景通常面临着数据量巨大及包含多种复杂的纹理噪声等问题。 本文针对大规模场景三维重建，提出了一种全向传感器设备的联合标定技术，并 基于该全向传感器系统提出了一个三维重建框架，旨在全面提升三维重建的效率 及纹理质量。 在传感器标定方面，本文提出的方法为自动化的全向传感系统标定方法，能 准确计算全景相机与全向雷达的内外参数，并为几何重建和纹理优化打下坚实基 础。实验结果表明，该标定方法相比起基于棋盘格的内参标定方法和基于互信息 的外参标定技术具有明显的精度优势。此外，本文还凭借标定好的传感器设备实 现了两级精度的建图，建图过程中的运动信息被运用于三维重建过程中。 在三维重建方面，采用体素哈希技术，高效对大规模场景进行重建，克服了传 统方法在数据处理速度和存储方面的限制。同时，提出了一种纹理优化方法，能 同时处理高光影响、帧间颜色不一致和物体遮挡等多种混合纹理噪声，显著提高 了重建模型的纹理细节和色彩质量。实验结果表明，该方法相较于传统的高光滤 除方法而言，更能保护图像饱和度和对比度，具有更好的处理质量。此外，整个框 架的处理能力可达到每秒处理一百万个点，展现了其在实时处理方面的巨大潜力。 本文所提出的全向传感器标定方法不仅能应用于三维重建领域，还能广泛应 用于同时定位与地图构建技术（Simultaneous Localization And Mapping，SLAM）等， 因而具备重要的推广应用价值。此外，本文所提出的三维重建框架展示了在测绘和 制图领域的应用潜力，特别是对于地理信息系统（Geographic Information System， GIS）的发展和文化遗产保护等领域的重要意义。因此，本文的研究成果不仅对学 术界有着重要的理论价值，也为实际应用提供了有效的技术支持和解决方案。

In recent years, with the development of the game modeling industry and the rise of applications such as virtual reality and augmented reality, 3D reconstruction technology has gradually become a hot topic of current research. The rapid iteration of sensor hard[1]ware has also made the rapid development of 3D reconstruction technology a reality. In real-world applications, large-scale scene reconstruction is often involved, which usually faces challenges such as massive data volumes and various complex texture noise. This paper proposes a joint calibration technique for omnidirectional sensor devices, and based on this omnidirectional sensor system, a 3D reconstruction framework is proposed, aiming to comprehensively improve the efficiency and texture quality of 3D reconstruction. In terms of sensor calibration, the method proposed in this paper is an automated calibration method for omnidirectional sensor system, which can accurately calculate the internal and external parameters of panoramic cameras and omnidirectional LiDAR, lay[1]ing a solid foundation for geometric reconstruction and texture optimization. Experimen[1]tal results show that this calibration method has obvious accuracy advantages compared to the chessboard-based intrinsic calibration method and the mutual information-based extrinsic calibration technique. Moreover, this paper also achieves two-level precision mapping with the calibrated sensor devices, where the motion information during the mapping process is utilized in the 3D reconstruction process. In terms of 3D reconstruction, voxel hashing technology is used to efficiently re[1]construct large-scale scenes, overcoming the limitations of traditional methods in data processing speed and storage. At the same time, a texture optimization method is pro[1]posed that can simultaneously handle various mixed texture noise such as highlight ef[1]fects, frame color inconsistency, and object occlusion, significantly improving the tex[1]ture details and color quality of the reconstructed model. Experimental results show that this method, compared to traditional highlight filtering methods, better preserves image saturation and contrast, offering better processing quality. Moreover, the processing ca[1]pacity of the entire framework can reach one million points per second, demonstrating its tremendous potential in real-time processing. The omnidirectional sensor calibration method proposed in this study can be applied not only in the field of 3D reconstruction but also widely in technologies such as Simultaneous Localization And Mapping, thus having significant potential for broad application. Additionally, the 3D reconstruction framework presented in this study demonstrates its application potential in the fields of surveying and mapping, particularly in the development of Geographic Information Systems and the protection of cultural heritage. Therefore, the research results of this paper not only have significant theoretical value for the academic community but also provide effective technical support and solutions for practical applications.

[1] 张彦雯, 胡凯, 王鹏盛. 三维重建算法研究综述.[J]. Journal of Nanjing University of Information Science & Technology (Natural Science Edition)/Nanjing Xinxi Gongcheng Daxue Xuebao (ziran kexue ban), 2020, 12(5).
[2] 胡芳侨, 黄永, 李惠. 建筑三维重建方法综述[J]. 智能建筑与智慧城市, 2020(5): 10-14.
[3] ZHOU Y, GALLEGO G, REBECQ H, et al. Semi-dense 3D reconstruction with a stereo event camera[C]//Proceedings of the European conference on computer vision (ECCV). 2018: 235- 251.
[4] SCHÖNBERGER J L, ZHENG E, FRAHM J M, et al. Pixelwise view selection for unstructured multi-view stereo[C]//Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14. Springer, 2016: 501-518.
[5] FRAHM J M, FITE-GEORGEL P, GALLUP D, et al. Building rome on a cloudless day[C]// Computer Vision–ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part IV 11. Springer, 2010: 368-381.
[6] SCHONBERGER J L, FRAHM J M. Structure-from-motion revisited[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 4104-4113.
[7] WANG Z, WU S, XIE W, et al. NeRF–: Neural radiance fields without known camera parameters[A]. 2021.
[8] 郑太雄, 黄帅, 李永福, 等. 基于视觉的三维重建关键技术研究综述[J]. 自动化学报, 2020, 46(4): 631-652.
[9] LINDNER M, KOLB A, HARTMANN K. Data-fusion of PMD-based distance-information and high-resolution RGB-images[C]//2007 International Symposium on Signals, Circuits and Systems: volume 1. IEEE, 2007: 1-4.
[10] KELLER M, LEFLOCH D, LAMBERS M, et al. Real-time 3d reconstruction in dynamic scenes using point-based fusion[C]//2013 International Conference on 3D Vision-3DV 2013. IEEE, 2013: 1-8.
[11] HAN J, SHAO L, XU D, et al. Enhanced computer vision with microsoft kinect sensor: A review[J]. IEEE transactions on cybernetics, 2013, 43(5): 1318-1334.
[12] WANG R, PEETHAMBARAN J, CHEN D. Lidar point clouds to 3-D urban models :: A review [J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2018, 11(2): 606-627.
[13] WANG R. 3D building modeling using images and LiDAR: A review[J]. International Journal of Image and Data Fusion, 2013, 4(4): 273-292.
[14] 王国珲, 钱克矛, 等. 线阵相机标定方法综述[J]. Acta Optica Sinica, 2020, 40(1): 0111011.
[15] BOUGUET J Y. Camera calibration toolbox for matlab[J]. http://www. vision. caltech. edu/bouguetj/calib_doc/, 2004.
[16] ZHANG Q, PLESS R. Extrinsic calibration of a camera and laser range finder (improves camera calibration)[C]//2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)(IEEE Cat. No. 04CH37566): volume 3. IEEE, 2004: 2301-2306.
[17] SCARAMUZZA D, HARATI A, SIEGWART R. Extrinsic self calibration of a camera and a 3d laser range finder from natural scenes[C]//2007 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 2007: 4164-4169.
[18] SCARAMUZZA D, IKEUCHI K. Omnidirectional camera[M]. Springer US, 2014.
[19] UNNIKRISHNAN R, HEBERT M. Fast extrinsic calibration of a laser rangefinder to a camera [J]. Robotics Institute, Pittsburgh, PA, Tech. Rep. CMU-RI-TR-05-09, 2005.
[20] YUAN C, LIU X, HONG X, et al. Pixel-level extrinsic self calibration of high resolution lidar and camera in targetless environments[J]. IEEE Robotics and Automation Letters, 2021, 6(4): 7517-7524.
[21] ZHANG X, ZHU S, GUO S, et al. Line-based automatic extrinsic calibration of lidar and camera [C]//2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021: 9347-9353.
[22] ZHU Y, LI C, ZHANG Y. Online camera-lidar calibration with sensor semantic information [C]//2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2020: 4970-4976.
[23] LOWE D G. Object recognition from local scale-invariant features[C]//Proceedings of the seventh IEEE international conference on computer vision: volume 2. Ieee, 1999: 1150-1157.
[24] BAY H, TUYTELAARS T, VAN GOOL L. Surf: Speeded up robust features[C]//Computer Vision–ECCV 2006: 9th European Conference on Computer Vision, Graz, Austria, May 7-13, 2006. Proceedings, Part I 9. Springer, 2006: 404-417.
[25] RUBLEE E, RABAUD V, KONOLIGE K, et al. ORB: An efficient alternative to SIFT or SURF [C]//2011 International conference on computer vision. Ieee, 2011: 2564-2571.
[26] SERAFIN J, GRISETTI G. Using augmented measurements to improve the convergence of ICP[C]//Simulation, Modeling, and Programming for Autonomous Robots: 4th International Conference, SIMPAR 2014, Bergamo, Italy, October 20-23, 2014. Proceedings 4. Springer, 2014: 566-577.
[27] LEFLOCH D, KLUGE M, SARBOLANDI H, et al. Comprehensive use of curvature for robust and accurate online surface reconstruction[J]. IEEE transactions on pattern analysis and machine intelligence, 2017, 39(12): 2349-2365.
[28] SARLIN P E, DETONE D, MALISIEWICZ T, et al. Superglue: Learning feature matching with graph neural networks[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 4938-4947.
[29] BESL P J, MCKAY N D. Method for registration of 3-D shapes[C]//Sensor fusion IV: control paradigms and data structures: volume 1611. Spie, 1992: 586-606.
[30] CHEN Y, MEDIONI G. Object modelling by registration of multiple range images[J]. Image and vision computing, 1992, 10(3): 145-155.
[31] RUSINKIEWICZ S, LEVOY M. Efficient variants of the ICP algorithm[C]//Proceedings third international conference on 3-D digital imaging and modeling. IEEE, 2001: 145-152.
[32] RUSINKIEWICZ S, HALL-HOLT O, LEVOY M. Real-time 3D model acquisition[J]. ACM Transactions on Graphics (TOG), 2002, 21(3): 438-446.
[33] IZADI S, KIM D, HILLIGES O, et al. Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera[C]//Proceedings of the 24th annual ACM symposium on User interface software and technology. 2011: 559-568.
[34] WHELAN T, KAESS M, FALLON M, et al. Kintinuous: Spatially extended kinectfusion[Z]. 2012.
[35] CHEN J, BAUTEMBACH D, IZADI S. Scalable real-time volumetric surface reconstruction [J]. ACM Transactions on Graphics (ToG), 2013, 32(4): 1-16.
[36] ZHOU Q Y, KOLTUN V. Dense scene reconstruction with points of interest[J]. ACM Transactions on Graphics (ToG), 2013, 32(4): 1-8.
[37] ZHOU Q Y, MILLER S, KOLTUN V. Elastic fragments for dense scene reconstruction[C]// Proceedings of the IEEE International Conference on Computer Vision. 2013: 473-480.
[38] DAI A, NIESSNER M, ZOLLHÖFER M, et al. Bundlefusion: Real-time globally consistent 3d reconstruction using on-the-fly surface reintegration[J]. ACM Transactions on Graphics (ToG), 2017, 36(4): 1.
[39] WHELAN T, SALAS-MORENO R F, GLOCKER B, et al. ElasticFusion: Real-time dense SLAM and light source estimation[J]. The International Journal of Robotics Research, 2016, 35(14): 1697-1716.
[40] KAZHDAN M, BOLITHO M, HOPPE H. Poisson surface reconstruction[C]//Proceedings of the fourth Eurographics symposium on Geometry processing: volume 7. 2006: 0.
[41] KAZHDAN M, CHUANG M, RUSINKIEWICZ S, et al. Poisson surface reconstruction with envelope constraints[C]//Computer graphics forum: volume 39. Wiley Online Library, 2020: 173-182.
[42] CHENG S W, DEY T K, SHEWCHUK J, et al. Delaunay mesh generation[M]. CRC Press Boca Raton, 2013.
[43] CURLESS B, LEVOY M. A volumetric method for building complex models from range images[C]//Proceedings of the 23rd annual conference on Computer graphics and interactive techniques. 1996: 303-312.
[44] FUHRMANN S, GOESELE M. Fusion of depth maps with multiple scales[J]. ACM Transactions on Graphics (TOG), 2011, 30(6): 1-8.
[45] ZENG M, ZHAO F, ZHENG J, et al. A memory-efficient kinectfusion using octree[C]// Computational Visual Media: First International Conference, CVM 2012, Beijing, China, November 8-10, 2012. Proceedings. Springer, 2012: 234-241.
[46] STEINBRÜCKER F, STURM J, CREMERS D. Volumetric 3D mapping in real-time on a CPU [C]//2014 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2014: 2021-2028.
[47] NIESSNER M, ZOLLHÖFER M, IZADI S, et al. Real-time 3D reconstruction at scale using voxel hashing[J]. ACM Transactions on Graphics (ToG), 2013, 32(6): 1-11.
[48] KÄHLER O, PRISACARIU V A, REN C Y, et al. Very high frame rate volumetric integration of depth images on mobile devices[J]. IEEE transactions on visualization and computer graphics, 2015, 21(11): 1241-1250.
[49] PRISACARIU V A, KÄHLER O, GOLODETZ S, et al. Infinitam v3: A framework for large scale 3d reconstruction with loop closure[A]. 2017.
[50] HE Y, KHANNA N, BOUSHEY C J, et al. Specular highlight removal for image-based di etary assessment[C]//2012 IEEE International Conference on Multimedia and Expo Workshops. IEEE, 2012: 424-428.
[51] YANG Q, WANG S, AHUJA N. Real-time specular highlight removal using bilateral filtering [C]//Computer Vision–ECCV 2010: 11th European Conference on Computer Vision, Herak lion, Crete, Greece, September 5-11, 2010, Proceedings, Part IV 11. Springer, 2010: 87-100.
[52] SHEN H L, ZHENG Z H. Real-time highlight removal using intensity ratio[J]. Applied optics, 2013, 52(19): 4483-4493.
[53] FU G, ZHANG Q, SONG C, et al. Specular Highlight Removal for Real-world Images[C]// Computer graphics forum: volume 38. Wiley Online Library, 2019: 253-263.
[54] YANG J, LIU L, LI S. Separating specular and diffuse reflection components in the HSI color space[C]//Proceedings of the IEEE International Conference on Computer Vision Workshops. 2013: 891-898.
[55] YAMAMOTO T, NAKAZAWA A. General improvement method of specular component separation using high-emphasis filter and similarity function[J]. ITE Transactions on Media Technology and Applications, 2019, 7(2): 92-102.
[56] WEI X, XU X, ZHANG J, et al. Specular highlight reduction with known surface geometry[J]. Computer Vision and Image Understanding, 2018, 168: 132-144.
[57] GUO X, CAO X, MA Y. Robust separation of reflection from multiple images[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2014: 2187-2194.
[58] JIN Y, LI R, YANG W, et al. Estimating reflectance layer from a single image: Integrating reflectance guidance and shadow/specular aware learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence: volume 37. 2023: 1069-1077.
[59] 郭圣逸, 李丽, 沈彬, 等. 基于多视角序列图像的高光去除 CycleGAN 网络.[J]. Journal of Zhengzhou University (Natural Science Edition), 2023, 55(5).
[60] LI W, GONG H, YANG R. Fast texture mapping adjustment via local/global optimization[J]. IEEE transactions on visualization and computer graphics, 2018, 25(6): 2296-2303.
[61] YE X, WANG L, LI D, et al. 3D reconstruction with multi-view texture mapping[C]//Neural Information Processing: 24th International Conference, ICONIP 2017, Guangzhou, China, November 14-18, 2017, Proceedings, Part III 24. Springer, 2017: 198-207.
[62] CHUANG M, LUO L, BROWN B J, et al. Estimating the Laplace-Beltrami operator by restricting 3d functions[C]//Computer graphics forum: volume 28. Wiley Online Library, 2009: 1475-1484.
[63] BERTALMIO M, SAPIRO G, CASELLES V, et al. Image inpainting[C]//Proceedings of the 27th annual conference on Computer graphics and interactive techniques. 2000: 417-424.
[64] DARABI S, SHECHTMAN E, BARNES C, et al. Image melding: Combining inconsistent images using patch-based synthesis[J]. ACM Transactions on graphics (TOG), 2012, 31(4): 1-10.
[65] HUANG J B, KANG S B, AHUJA N, et al. Image completion using planar structure guidance [J]. ACM Transactions on graphics (TOG), 2014, 33(4): 1-10.
[66] MERTENS T, KAUTZ J, VAN REETH F. Exposure Fusion[C/OL]//15th Pacific Conference on Computer Graphics and Applications (PG’07). 2007: 382-390. DOI: 10.1109/PG.2007.17.
[67] VINCENT O R, FOLORUNSO O, et al. A descriptive algorithm for sobel image edge detection [C]//Proceedings of informing science & IT education conference (InSITE): volume 40. 2009: 97-107.
[68] WANG X. Laplacian operator-based edge detectors[J]. IEEE transactions on pattern analysis and machine intelligence, 2007, 29(5): 886-890.
[69] DING L, GOSHTASBY A. On the Canny edge detector[J]. Pattern recognition, 2001, 34(3): 721-725.
[70] XIA M, YAO J, XIE R, et al. Color consistency correction based on remapping optimization for image stitching[C]//Proceedings of the IEEE international conference on computer vision workshops. 2017: 2977-2984.
[71] PANDEY G, MCBRIDE J, SAVARESE S, et al. Automatic targetless extrinsic calibration of a 3d lidar and camera by maximizing mutual information[C]//Proceedings of the AAAI Conference on Artificial Intelligence: volume 26. 2012: 2053-2059.
[72] TERRELL G R, SCOTT D W. Variable kernel density estimation[J]. The Annals of Statistics, 1992: 1236-1265.
[73] SEGAL A, HAEHNEL D, THRUN S. Generalized-icp.[C]//Robotics: science and systems: volume 2. Seattle, WA, 2009: 435.
[74] XU W, ZHANG F. Fast-lio: A fast, robust lidar-inertial odometry package by tightly-coupled iterated kalman filter[J]. IEEE Robotics and Automation Letters, 2021, 6(2): 3317-3324.
[75] HORE A, ZIOU D. Image quality metrics: PSNR vs. SSIM[C]//2010 20th international conference on pattern recognition. IEEE, 2010: 2366-2369.
[76] ZHANG L, ZHANG L, MOU X, et al. FSIM: A feature similarity index for image quality assessment[J]. IEEE transactions on Image Processing, 2011, 20(8): 2378-2386.