Spring and Fall Semester Implementation<\/h2>\n\n\n\n
The following description about our subsystem implementation is distributed into two broad time periods: the Spring semester and the Fall semester. The fire localization and the state estimation subsystems were of major focus in the spring semester, however in the interest of a successful demonstration and realism to our use case, we took decision in the fall semester which extensively changed our subsystem design as compared to the spring semester including designing a new drone. To explain this aptly, we first describe the implementation summary of the fall semester, referring to the spring semester implementation details along the way. The spring semester work would be displayed below the fall semester work.<\/p>\n\n\n\n
Fall Semester<\/h2>\n\n\n\n[Revamped] Fire Perception<\/h2>\n\n\n\n
Fire Localization in Phoenix:<\/strong> In our previous aerial system, Phoenix, the fire localization module processed thermal images to determine fire hotspot coordinates for mapping. Two methods were explored: a learning-based approach developed by AirLab and a classical method developed by our team.<\/p>\n\n\n\n The deployed method relied on classical stereo matching of synchronized thermal feeds, rectified using OpenCV’s stereoRectify<\/em> to correct distortions. Temperature-based masks were generated to segment hotspots, and ORB features were then detected and matched between the left and right frames. The matches were refined using epipolar constraints to estimate depth and compute 3D coordinates in the world frame.<\/p>\n\n\n\n Alternative approaches included generating disparity maps with StereoSGBM and using Fast-ACVNet for learning-based depth estimation. These methods showed promise but faced challenges such as information loss during image conversion and poor performance due to distribution mismatches.<\/p>\n\n\n\n While Phoenix Pro employs a different fire localization system, these efforts informed our understanding of thermal-based perception.<\/p>\n\n\n\n In the Phoenix Pro<\/em> system,<\/strong> the fire perception module was designed to localize and map fire hotspots more accurately compared to the previous approach used in Phoenix<\/em>. The previous system relied on a thermal stereo-based design, which exhibited a localization error of approximately 2 meters within a range of 5 meters. This approach faced several challenges, including the requirement for a large baseline between thermal cameras and the difficulty of performing precise extrinsic calibration between them. These factors contributed to inaccuracies in fire localization, limiting the effectiveness of the system for real-world applications.<\/p>\n\n\n\n To overcome these challenges and improve localization accuracy, we adopted a new approach in Phoenix Pro<\/em> that leveraged the RealSense point cloud, effectively bypassing the need for thermal stereo. The RealSense camera provided dense RGB-D data, enabling us to obtain accurate depth measurements without the need for large baseline configurations or complex thermal camera calibration. By integrating thermal information with the RealSense point cloud, we could project fire hotspots detected in the thermal images into 3D space with higher precision. This method significantly enhanced the accuracy, localizing fires now up to 50 cm error for a 6 m range.<\/p>\n\n\n\n The overall flow of information from the sensors (FLIR Boson + RealSense D456) to the fire localization and mapping is depicted in the figure below. We receive data from FLIR Boson at 30 fps, and point cloud from RS D456 at 30 fps. We first downsample the point cloud with a leaf size of 0.1 m, and clip the range to 8 m. This reduces the computational load and filters out bad points in the point cloud. With this configuration, we are able to achieve localization at 30 fps, i.e., in real time.<\/p>\n\n\n We do a binary segmentation of fire hotspots, transform the point cloud into FLIR Boson’s optical frame using RS-Thermal extrinsics, and then project the points onto the FLIR’s frame using the intrinsic parameters of the FLIR. We reproject only the pixels corresponding to the hotspots using the mask, and then publish the centroid of the reprojected points. For tackling multiple hotspots, we cluster the segments in the binary mask generated from the thermal image using OpenCV’s Note that the range of the point cloud affects the fire perception capability during exploration. This parameter should be set based on the depth range assigned in the exploration. If the drone performs less exploration, then it is advised to set a longer range. However, this would affect localization accuracy due to poor points from the RS D456.<\/p>\n\n\n\n Similar to the Spring semester, the global fire mapping subsystem handles the temporal side of the fire perception subsystem. The fire perception module provides raw measurements of hotspot locations relative to the world frame. When the fire perception subsystem publishes a new set of measurements, this mapping subsystem is executed as shown in the flowchart above.<\/p>\n\n\n\n Note<\/strong>: Unlike the Spring where we had a dedicated filtering module, the introduction of RGBD-Thermal fusion significantly enhances the quality of our raw measurements. Specifically, moving away from the triangulation approach, there are now no false positives behind the camera.<\/em><\/mark><\/p>\n\n\n\n Adding New Hotspot<\/strong>: We then perform nearest neighbor clustering for each hotspot to determine whether it is a measurement for a new or existing one. A new hotspot is created if it exceeds distances from all previously seen hotspots.<\/p>\n\n\n\n Updating Existing Hotspot<\/strong>: If we instead find any measurement corresponding to an existing hotspot, we add it to the existing nearest-neighbor map, and use a mean search to find the updated position of its parent hotspot.<\/p>\n\n\n\n In line with our non-functional requirement MNF4<\/strong>, which mandates the use of passive sensing modalities, and considering the GPS-denied environment in which our system operates, we developed the state-estimation module to use two NIR cameras and an IMU sensor as inputs to estimate visual odometry.<\/p>\n\n\n\n We choose to use off-the-shelf VINS Fusion<\/a> which is an optimization-based multi-sensor state estimation package due to its wide adoption in the robotics community and its recognition for robustness. <\/p>\n\n\n\n The key prerequisite for using the package with an IMU and cameras is to provide accurate IMU-to-camera extrinsic, camera intrinsics, IMU gyroscope and accelerometer biases (noise), and the time offset between the IMU and camera topics. The figure below shows the calibration pipeline developed to accomplish these tasks. Initially, Kalibr<\/a> calibration package is used to obtain the camera intrinsics for the RealSense left and right NIR cameras using a checkerboard pattern. The imu_variance_ros<\/a> package was then used to estimate the IMU biases, which were subsequently used by Kalibr to perform the calibration and obtain the IMU-to-camera extrinsic (for both cameras) and time offset.<\/p>\n\n\n\n The flow of our state estimation module is illustrated below. The VINS node uses an IMU sensor and two NIR (Near-Infrared) images from the left and right cameras of the RealSense D456 to estimate the odometry. This data is then processed by the Transformer node, which publishes the odometry at a fixed rate of 25Hz, transformed with respect to the UAV platform’s base-link frame. Additionally, this node is also responsible for broadcasting the transforms between the map, IMU, odometry, base-link, and thermal camera to the RealSense depth frame.<\/p>\n\n\n Note: <\/em><\/strong>In the Fall, we decided not to use Multi-Spectral Odometry (MSO), previously used in our Spring Demonstration. The decision was based on its limited documentation, tight integration within a Docker environment on ARM-based hardware (ORIN on ORDv1), incomplete code commits, and the departure of its key developer from Airlab, making it difficult to integrate into the new platform (Phoenix Pro)<\/strong>.<\/em><\/p>\n\n\n\nFusing RGBD and Thermal<\/h3>\n\n\n\n
![\"\"](\"https:\/\/mrsdprojects.ri.cmu.edu\/2024teamb\/wp-content\/uploads\/sites\/75\/2024\/12\/fire_perception_pipeline-1024x321.png\")
<\/figure>\n<\/div>\n\n\nconnected_components<\/code>. Clustering in 2D image space rather than 3D space reduces the computational overload significantly and enables real-time operation. This is depicted in the following figure.<\/p>\n\n\n\n![\"\"](\"http:\/\/mrsdprojects.ri.cmu.edu\/2024teamb\/wp-content\/uploads\/sites\/75\/2024\/12\/fire_pipe-1024x356.png\")
<\/figure>\n\n\n\nGlobal Fire Mapping<\/h3>\n\n\n\n
![\"\"](\"https:\/\/mrsdprojects.ri.cmu.edu\/2024teamb\/wp-content\/uploads\/sites\/75\/2024\/12\/fire_mapping_flowchart-1024x319.png\")
<\/figure>\n\n\n\n
\n<\/div>\n<\/div>\n\n\n\n[Revamped] State Estimation<\/h2>\n\n\n\n
VINS Fusion<\/h3>\n\n\n\n
![\"\"](\"https:\/\/mrsdprojects.ri.cmu.edu\/2024teamb\/wp-content\/uploads\/sites\/75\/2024\/12\/vins-calibration-pipeline-1.png\")
<\/figure>\n\n\n\nOverall Pipeline<\/h3>\n\n\n\n
![\"\"](\"https:\/\/mrsdprojects.ri.cmu.edu\/2024teamb\/wp-content\/uploads\/sites\/75\/2024\/12\/state-estimation-flow-1.png\")
<\/figure>\n<\/div>\n\n\n
\n\n\n\nState Manager<\/h2>\n\n\n\n
![\"\"](\"https:\/\/mrsdprojects.ri.cmu.edu\/2024teamb\/wp-content\/uploads\/sites\/75\/2024\/12\/autonomy_plan-1024x278.png\")