Figure 1.
Visualization of the system including RGB cameras, thermal cameras, and LiDAR. 360 RGB camera and 360 thermal camera are made from independent cameras to remove blind spots. Images and point clouds are compensated to decrease negative impacts of motion. Then, point clouds and images are used for sensor calibration based on extracted features.

Figure 2.
Visualization of the target detected by two types of cameras. The (left image) is the target detected by the RGB camera and the (right image) is the target detected by the thermal camera.

Figure 3.
Our system includes sensors: LiDAR Velodyne Alpha Prime, LadyBug-5 camera, 6 FLIR ADK cameras, LiDAR Ouster-128, LiDAR Ouster-64 and LiDAR Hesai Pandar.

Figure 4.
Visualization of stitching 360 thermal images and 360 RGB images.

Figure 5.
Pipeline of the registration process. The approach is divided into two parts, one part focuses on detecting key points from RGB images and thermal images, while the other part detects key points from images converted from LiDAR point clouds. For images generated from LiDAR point clouds, a velocity estimation step is required to perform distortion correction, ensuring the accurate positioning of the scanned points. After getting results from distortion correction, external parameters of LiDAR, 360 RGB camera and 360 thermal camera can be calibrated.

Figure 6.
Visualization of features extracted from RGB images.

Figure 7.
Pipeline of our approach. The first step is enhancing images by Retinex Decomposition. The second step is to extract key features from $n+1$ consecutive images. The third step is using MobileNetV3 to remove noise features on moving objects.

Figure 8.
The (above image) shows results before being enhanced by Retinex Decomposition. The (below image) shows results after being enhanced by Retinex Decomposition.

Figure 9.
The (above image) including the red rectangles shows reliable features extracted from $n+1$ consecutive RGB images. The (below image) including the green rectangles shows reliable features after filtering by MobileNetV3.

Figure 10.
Visualization of features extracted from thermal images.

Figure 11.
Visualization of image projection. (a) shows the 3D point cloud data from the LiDAR. (b) presents the 2D image data with the intensity channel. (c) presents the 2D image data with the range channel. The height of the image is 128, corresponding to the number of channels in the LiDAR.

Figure 12.
Visualization of key points extracted from LiDAR images. (a) simulates key points across two frames, while (b) simulates selecting key points with similarity across the two frames.

Figure 13.
Pipeline of our approach. Key features of projected images are extracted by Superpoint enhanced by LSTM. These features are matched to find pair points in two consecutive frames.

Figure 14.
Pipeline of the ego-motion compensation process. First, the point clouds are converted into two-dimensional images using Spherical Projection. Key features are then identified within these range images, and corresponding point pairs are matched. By matching key feature pairs, the distance between frames can be determined, allowing for velocity estimation. Finally, velocity and timestamp will be used to resolve ego-motion compensation and point cloud accumulation.

Figure 15.
Visualization of distortion correction. The motion of the vehicle is presented by the circles, and the LiDAR is also sotating while the vehicle is in motion. (a) shows the actual shape of the obstacle. (b) depicts the shape of the obstacle scanned by LiDAR. (c) illustrates the shape of the obstacle after distortion correction.

Figure 16.
Visualization of the differences in distortion correction on 3D point clouds within a frame with a speed of 54 km/h and a frequency of 10 Hz. The red part shows the original points of the point clouds, while the green part shows the corrected points. The left image shows points on the $xy$-plane. The right image shows points on the $yz$-plane.

Figure 17.
Visualization of distortion correction of cameras. The blue rectangle is the actual shape, and the red rectangle is the shape distorted by ego-motion.

Figure 18.
Visualization of 360 RGB–LiDAR images calibration. The (above image) including the red rectangles indicate the calibration results before applying correction. The (below image) including the green rectangles indicates the calibration results after applying correction.

Figure 19.
Visualization of 360 RGB–thermal images calibration. The (above image) includes red rectangles that indicate the calibration results before applying correction. The (below image) includes green rectangles that indicate the calibration results after applying correction.

Figure 20.
Visualization of point clouds extracted from Ouster OS1-128 and Velodyne Alpha prime.

Figure 21.
Visualization of image projection. (a) presents the 2D image data with the intensity channel from Ouster OS1-128. (b) presents the 2D image data with the range channel from Ouster OS1-128. (c) presents the 2D image data with the intensity channel from Velodyne Alpha prime. (d) presents the 2D image data with the range channel from Velodyne Alpha prime.

Figure 23.
Comparison with CNN, RIFT, RI-MFM by MAE.

Figure 24.
Comparison with CNN, RIFT, RI-MFM by Accuracy.

Figure 25.
Comparison with CNN, RIFT, RI-MFM by RMSE.

Figure 26.
Red points represent the results of calibration without distortion correction, while blue points represent the results with distortion correction in static situations. The dashed line is the results from the target based method.

Figure 27.
Red points and blue points represent the results of calibration without and with distortion correction in dynamic situations. The dashed lines present the results using the actual data.

Figure 28.
Comparison of error in rotation and translation of three methods.