Online Camera-LiDAR Calibration with 3D Photometric Consistency

In recent times, the fusion of three-dimensional Light Detection and Ranging (LiDAR) alongside imaging devices has become increasingly common in the realm of self-driving cars and autonomous robotics. LiDAR sensors are widely employed due to their excellent 3D ranging abilities in tasks such as area mapping, tracking objects, avoiding obstacles, and other activities where range is crucial. As a rule, such sensors feature a reduced angular resolution in contrast to cameras. However, despite their usual high angular resolution, capturing range data is a challenge for cameras. This problem occurs because monocular cameras omit range information in the imaging process. The quality of range data obtained through triangulation techniques using multiple cameras usually falls short compared to LiDAR, especially in expansive outdoor environments. Consequently, the combination of three-dimensional LiDAR data and two-dimensional camera data is considered a compelling method across a wide range of applications, by leveraging the combined advantages of both cameras and LiDAR sensors. Regarding the fusion of sensor data, extrinsic calibration is of paramount importance. The content of this paper introduces a method based on uniformity in lighting and color characteristics for detecting and correcting misalignments between camera and LiDAR across different environments, in real-time, without the need for calibration tools or human interference. This paper assumes that with accurate external parameters and proper estimation of LiDAR positioning, every projected point from the LiDAR onto different camera images will exhibit similar lighting and color values. The proposal involves using covisibility data to establish an error term grounded in the previously mentioned photometric consistency concept, facilitating the identification and rectification of calibration errors. A series of experiments with real-world data sequences were carried out, demonstrating the effectiveness of the suggested error term in detecting and rectifying miscalibrations.

近年来，三维激光雷达点云数据与二维相机图像数据的融合在移动机器人与自动驾驶领域逐渐变得普遍。其中，激光雷达具有良好的3D测距能力，被广泛用于建图、避障、物体追踪等距离测量较为重要的任务场景中。相机作为一种二维图像传感器，其角分辨率远高于激光雷达，可高效的获取视野内物体表面的纹理信息，但无法有效的获取距离信息。因为使用单目相机时，距离信息在成像过程中被丢弃了。尽管可以使用多目相机，基于三角测量来恢复距离信息，但这种方式获取的距离质量通常低于激光雷达的测量质量，特别是在大型户外场景中。因此，激光雷达与相机的数据融合逐渐成为了各种应用场景中具有较强吸引力的解决方案，而激光雷达与相机硬件的外参校准问题是传感器数据融合的重中之重。本文介绍了一种基于时空光度一致性的方法，用于在线检测误标定并提供实时的重新标定，无需标定靶或手动的数据关联工作。 本文利用了光度一致性假设：在拥有正确的外参和准确的激光雷达位姿估计时，单个激光雷达的数据点在不同相机图像上的投影将具有相似的光度值。 基于该假设，本文提出了一个能够用于检测和校正错误标定的误差项，并使用真实数据集进行了多次实验，证明该误差项在多个场景中均有较好的表现，即能够及时感知误标定的存在，并提供实时的重新标定。

[1] J.Zhang and S. Singh,“LOAM:Lidar odometry and mapping in real-time,” in Proc. Robot.: Sci. Syst. Conf., Jan. 2014, pp. 109–111.
[2] K. Cho, S. Baeg, and S. Park, “Real-time 3D multiple occluded object detection and track-ing,” in Proc. IEEE ISR, 2013, pp. 1–5.
[3] Y. Peng, D. Qu,Y. Zhong, S. Xie, J. Luo, and J.Gu, “The obstacle detection and obstacle avoidance algorithm based on 2-D LiDAR,” in Proc. IEEE Int. Conf. Inf. Automat., 2015, pp. 1648–1653.
[4] N. Li and B. Su, “3D-LiDAR based obstacle detection and fast map reconstruction in rough terrain,” in Proc. IEEE 5th Int. Conf. Automat. Control Robot. Eng., 2020, pp. 145–151.
[5] L. Xiao, R.Wang, B. Dai, Y. Fang, D. Liu, and T.Wu, “Hybrid conditional random field based camera-LiDAR fusion for road detection,” Inf. Sci., vol. 432, no. 8, pp. 543–558, 2017.
[6] Y. Zhu, C. Zheng, C. Yuan, X. Huang, and X. Hong, “Camvox: A low-cost and accurate LiDAR-assisted visual SLAM system,” in Proc. IEEE Int. Conf. Robot. Automat., 2021, pp. 5049–5055.
[7] J. Zhang and S. Singh, “Visual-LiDAR odometry and mapping: Low-drift, robust, and fast,” in Proc. IEEE Int. Conf.Robot.Automat., 2015, pp. 2174–2181.
[8] L. Zhou, Z. Li, and M. Kaess, “Automatic extrinsic calibration of a camera and a3DLiDARusing line and plane correspondences,” in Proc. IEEE/RSJ Int. Conf. Intell. Ro-bots Syst., 2018, pp. 5562–5569.
[9] J. Cui, J. Niu, Z. Ouyang, Y. Q. He, and D. Liu, “ACSC: Automatic calibration for non-repetitive scanning solid-state LiDAR and camera systems,” 2020, arXiv:2011.08516.
[10] E. S. Kim and S. Y. Park, “Extrinsic calibration between camera and LiDAR sensors by matching multiple 3D planes,” Sensors (Switzerland), vol. 20, no. 1, 2020, Art. no. 52.
[11] C. Yuan, X. Liu, X. Hong, and F. Zhang, “Pixel-level extrinsic self calibration of high resolution LiDAR and camera in targetless environments,” IEEE Robot. Automat. Lett., vol. 6, no. 4, pp. 7517–7524, Oct. 2021.
[12] J. Levinson and S. Thrun, “Automatic online calibration of cameras and lasers,” Robot.: Sci. Syst., vol. 2, 2013. [Online]. Available: https://www.semanticscholar.org/paper/Automatic-Online-Calibration-of-Camerasand-Lasers-Levinson-Thrun/73bed33a5aa661b183ae042783c9ccff2c5820df
[13] Livox- SDK, “Livox camera LiDAR calibration,” 2020. [Online]. Available: https://github.com/Livox-SDK/livox_camera_LIDAR_calibration
[14] X. Gong,Y. Lin, and J. Liu, “3D LiDAR-camera extrinsic calibration using an arbitrary trihedron,” Sensors, vol. 13, pp. 1902–1918, 2013.
[15] J. Kümmerle and T. Kühner, “Unified intrinsic and extrinsic camera and Li-DAR calibration under uncertainties,” in Proc. IEEE Int. Conf. Robot. Auto-mat., 2020, pp. 6028–6034.
[16] Y. Park, S. Yun, C.Won, K. Cho, K. Um, and S. Sim, “Calibration between color camera and 3D LiDAR instruments with a polygonal planar board,” Sen-sors, vol. 14, pp. 5333–5353, 2014.
[17] G. Koo, J. Kang, B. Jang, and N. Doh, “Analytic plane covariances construc-tion for precise planarity-based extrinsic calibration of camera and LiDAR,” in Proc. IEEE Int. Conf. Robot. Automat., 2020, pp. 6042–6048.
[18] B. Nagy, L. Kovács, and C. Benedek, “Online targetless end-to-end camera-LiDAR self-calibration,” in Proc. IEEE 16th Int. Conf. Mach. Vis. Appl., 2019, pp. 1–6.
[19] R. Ishikawa, T. Oishi, and K. Ikeuchi, “LiDAR and camera calibration using motions estimated by sensor fusion odometry,” in Proc. IEEE Int. Conf. Intell. Robots Syst., 2018, pp. 7342–7349.
[20] P. Biasutti, A. Bugeau, J.-F. Aujol, andM. Brédif, “Visibility estimation in point clouds with variable density,” in Proc. Int. Conf. Comput. Vis. Theory Appl., 2019. [Online]. Available: https://www.semanticscholar.org/paper/Visibility-Estimation-in-Point-Clouds-with-Variable-Biasutti-Bugeau/158baecc7bf9f083c1c1ffb2c67ba2d24b1ba33f
[21] Y. Zhao and P. A. Vela, “Good feature matching: Toward accurate, robust VO/VSLAM with low latency,” IEEE Trans. Robot., vol. 36, no. 3, pp. 657–675, Jun. 2020.
[22] S. Agarwal et al., “Ceres solver,” Version 2.2, Oct. 2023. [Online]. Available: https://github.com/ceres-solver/ceres-solver
[23] A. Geiger, P. Lenz, andR.Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2012, pp. 3354–3361.
[24] Z. Taylor and J. Nieto, “Motion-based calibration of multimodal sensor extrin-sics and timing offset estimation,” IEEE Trans. Robot., vol. 32, no. 5, pp. 1215–1229, Oct. 2016.
[25] Y. Pan, P. Xiao, Y. He, Z. Shao, and Z. Li, “MULLS: Versatile LiDAR SLAM via multi-metric linear least square,” in Proc. IEEE Int. Conf.Robot. Automat., 2021. pp. 11633–11640.
[26] C. Yuan, W. Xu, X. Liu, X. Hong, and F. Zhang, “Efficient and probabilistic adaptive voxel mapping for accurate online LiDAR odometry,” IEEE Robot. Automat. Lett., vol. 7, no. 3, pp. 8518–8525, 2022, doi: 10.1109/LRA.2022.3187250.
[27] C. Zheng, Q. Zhu, W. Xu, X. Liu, Q. Guo, and F. Zhang, “FAST-LIVO: Fast and tightly-coupled sparse-direct LiDAR-inertial-visual odometry,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst., 2022, pp. 4003–4009, doi: 10.1109/IROS47612.2022.9981107.
[28] C. Liu, J. Xiao, L. Lv, G. Xu, Z. Feng and J. Ge, "Extrinsic Calibration be-tween Camera and LiDAR Sensors by Virtual Planar Junctions Matching," 2020 7th International Conference on Information Science and Control Engineering (ICISCE), Changsha, China, 2020, pp. 597-601, doi: 10.1109/ICISCE50968.2020.00129.
[29] Z. Chai, Y. Sun and Z. Xiong, "A Novel Method for LiDAR Camera Calibra-tion by Plane Fitting," 2018 IEEE/ASME International Conference on Ad-vanced Intelligent Mechatronics (AIM), Auckland, New Zealand, 2018, pp. 286-291, doi: 10.1109/AIM.2018.8452339.
[30] Y. Yao, H. Xiaoyan and L. Jiliang, "A Space Joint calibration method for lidar and camera on self-driving car and its experimental verification," 2021 6th In-ternational Symposium on Computer and Information Processing Technology (ISCIPT), Changsha, China, 2021, pp. 388-394, doi: 10.1109/ISCIPT53667.2021.00084.
[31] S. Mishra, G. Pandey and S. Saripalli, "Extrinsic Calibration of a 3D-LIDAR and a Camera," 2020 IEEE Intelligent Vehicles Symposium (IV), Las Vegas, NV, USA, 2020, pp. 1765-1770, doi: 10.1109/IV47402.2020.9304750.
[32] W. Zhang and D. Xu, "Extrinsic Calibration of LiDAR-Camera Based on Deep Convolutional Network," 2022 China Automation Congress (CAC), Xiamen, China, 2022, pp. 2949-2954, doi: 10.1109/CAC57257.2022.10055799.
[33] Y. Li et al., "Application of 3D-LiDAR & Camera Extrinsic Calibration in Ur-ban Rail Transit," 2020 IEEE 5th International Conference on Intelligent Transportation Engineering (ICITE), Beijing, China, 2020, pp. 456-460, doi: 10.1109/ICITE50838.2020.9231446.
[34] L. Wang, Z. Xiao, D. Zhao, T. Wu and B. Dai, "Automatic Extrinsic Calibra-tion of Monocular Camera and LIDAR in Natural Scenes," 2018 IEEE Interna-tional Conference on Information and Automation (ICIA), Wuyishan, China, 2018, pp. 997-1002, doi: 10.1109/ICInfA.2018.8812555.
[35] Y. Lyu, L. Bai, M. Elhousni and X. Huang, "An Interactive LiDAR to Camera Calibration," 2019 IEEE High Performance Extreme Computing Conference (HPEC), Waltham, MA, USA, 2019, pp. 1-6, doi: 10.1109/HPEC.2019.8916441.
[36] L. Yan, J. Dai, X. Hu, Y. Li, S. Su and H. Xie, "Automatic Targetless Extrin-sic Calibration between a Spinning Actuated LiDAR and a Camera," 2022 IEEE International Conference on Unmanned Systems (ICUS), Guangzhou, China, 2022, pp. 303-308, doi: 10.1109/ICUS55513.2022.9987130.
[37] S. Fan, Y. Yu, M. Xu and L. Zhao, "High-Precision External Parameter Cali-bration Method for Camera and Lidar Based on a Calibration Device," in IEEE Access, vol. 11, pp. 18750-18760, 2023, doi: 10.1109/ACCESS.2023.3247195.
[38] D. Lee and S. -C. Kee, "Efficient Camera–LiDAR Calibration Using Accumu-lated LiDAR Frames," in IEEE Access, vol. 10, pp. 132349-132362, 2022, doi: 10.1109/ACCESS.2022.3230463.
[39] X. Liu, C. Yuan and F. Zhang, "Targetless Extrinsic Calibration of Multiple Small FoV LiDARs and Cameras Using Adaptive Voxelization," in IEEE Transactions on Instrumentation and Measurement, vol. 71, pp. 1-12, 2022, Art no. 8502612, doi: 10.1109/TIM.2022.3176889.
[40] N. Ou, H. Cai and J. Wang, "Targetless Lidar-Camera Calibration via Cross-Modality Structure Consistency," in IEEE Transactions on Intelligent Vehicles, vol. 9, no. 1, pp. 2636-2648, Jan. 2024, doi: 10.1109/TIV.2023.3337490.
[41] A. Rashd, W. Hardt, A. Kolker, M. Bdiwi and M. Putz, "Open-Box Target for Extrinsic Calibration of LiDAR, Camera and Industrial Robot," 2020 3rd In-ternational Conference on Mechatronics, Robotics and Automation (ICMRA), Shanghai, China, 2020, pp. 121-125, doi: 10.1109/ICMRA51221.2020.9398362.
[42] Y. Cai, Y. Zhan and W. Deng, "A Novel Extrinsic Calibration Method of a Camera-And-LiDAR System," 2021 IEEE 7th International Conference on Vir-tual Reality (ICVR), Foshan, China, 2021, pp. 109-116, doi: 10.1109/ICVR51878.2021.9483866.
[43] Y. Zhao, K. Huang, H. Lu and J. Xiao, "Extrinsic Calibration of a Small FoV LiDAR and a Camera," 2020 Chinese Automation Congress (CAC), Shanghai, China, 2020, pp. 3915-3920, doi: 10.1109/CAC51589.2020.9327398.
[44] B. Fu, Y. Wang, X. Ding, Y. Jiao, L. Tang and R. Xiong, "LiDAR-Camera Calibration Under Arbitrary Configurations: Observability and Methods," in IEEE Transactions on Instrumentation and Measurement, vol. 69, no. 6, pp. 3089-3102, June 2020, doi: 10.1109/TIM.2019.2931526.
[45] A. Singandhupe, H. M. La and Q. P. Ha, "Single Frame Lidar-Camera Calibra-tion Using Registration of 3D Planes," 2022 Sixth IEEE International Confer-ence on Robotic Computing (IRC), Italy, 2022, pp. 395-402, doi: 10.1109/IRC55401.2022.00076.
[46] A. Rashd, W. Hardt, A. Kolker, M. Bdiwi and M. Putz, "Open-Box Target for Extrinsic Calibration of LiDAR, Camera and Industrial Robot," 2020 3rd In-ternational Conference on Mechatronics, Robotics and Automation (ICMRA), Shanghai, China, 2020, pp. 121-125, doi: 10.1109/ICMRA51221.2020.9398362.
[47] H. Liu, Q. Xu, Y. Huang, Y. Ding and J. Xiao, "A Method for Synchronous Automated Extrinsic Calibration of LiDAR and Cameras Based on a Circular Calibration Board," in IEEE Sensors Journal, vol. 23, no. 20, pp. 25026-25035, 15 Oct.15, 2023, doi: 10.1109/JSEN.2023.3312322.
[48] Y. Jing, C. Yuan and X. Hong, "Online Calibration Between Camera and Li-DAR With Spatial-Temporal Photometric Consistency," in IEEE Robotics and Automation Letters, vol. 9, no. 2, pp. 1027-1034, Feb. 2024, doi: 10.1109/LRA.2023.3341768.