ROS-Based Calibration and Fusion of Object-Capturing and Referencing Sensors in MSS Approach

Multi-sensor systems (MSSs) have recently been used widely as a significant technological advancement in various fields e.g., environmental monitoring, robotic and autonomous driving. By enabling collection and integration of data from multiple sources simultaneously and use the capabilities of different sensors, such as cameras, light detection and ranging (LiDAR), global navigation satellite system (GNSS) and inertial measurement unit (IMU), MSSs provide a better and accurate understanding of the surrounding environment, better decision-making, and enhance efficiency.

Nevertheless, implementing a MSS presents several challenges, that include data capturing, synchronization, sensor calibration, and data fusion. While many projects and tools that handle these issues were developed, they often focus on an individual problem, use different format types of inputs and outputs, and rely on a specific environments and dependencies, which make it hard to integrate them together.

To address these challenges, this work proposes a comprehensive workflow that covers the essential functionalities required by a MSS. The work ranges from capturing data in real-world environment to calibration, synchronization, filtering, and fusion.

Special emphasis will be placed on camera calibration, LiDAR -Camera calibration, and GNSS-IMU fusion. Camera calibration primarily aims to determine the intrinsic parameters of the camera, which serve to correct for lens distortion and measure object size in world units. Whereas LiDAR -Camera calibration will be used for the estimation of the extrinsic parameters of the LiDAR sensor and camera, involving location and orientation, to establish relative rotation and translation between both sensors. The solution of GNSS-IMU fusion will be based on error-state Kalman filter (ESKF), which is used as a navigation filter for state estimation by estimating the error of the true system state rather than directly estimating the true state itself.

The proposed workflow will be implemented using the Robot Operating System (ROS), a framework that uses a node-based architecture and employs a publish-subscribe communication model. This framework enables flexibility and scalability in system design and resolve the dependency issue. Moreover, it supports a wide range of hardware platforms, and provides a set of powerful tools and utilities that support development and visualization.