Software Design and Implementation

Software CodeBase

Our codebase can be found at : [] with mainly two components: the Multi-spectral Odometry and the Fire localization and mapping modules.

Subsystems

Depth Estimation

Implementation (scheduled for FVD)

Fire Perception

Implementation [Exploration Phase]

Fire Segmentation: The fire segmentation pipeline processes raw radiometric readings by both left and right cameras, and converts them into easily readable temperature values, typically measured in Celsius. Following this conversion, a binary segmentation pipeline is used to isolate the fire hotspot region by masking out all pixels below a predetermined threshold value, effectively highlighting the area of interest. Additionally, another pipeline is available to conduct slab-wise segmentation of the temperature map, allowing for a more detailed analysis.
Fire Localization: In the development of the fire localization system, several methods were explored but encountered various challenges. The first approach utilized the OpenCV StereoSGBM module to generate disparity maps from thermal stereo images. This method faced issues with accuracy due to the necessary conversion of 16-bit images to 8-bit, resulting in significant information loss (image smoothing and noisy disparity maps). An enhancement was attempted by performing image processing of the rectified thermal images; however, this improvement yielded minimal enhancement in the feature-matching process. Lastly, a learning-based method using a Fast-ACVnet architecture was trialled, which showed promising results with disparity maps achieving a performance of 13 frames per second. Despite some artefacts, the depth estimation near the ground was reliable, suggesting potential for practical application in fire mapping. Although the results from the learned model were good, this approach can not be used for our application since the space heaters that were used for simulating fire were out of the distribution of the model. Hence the learning-based model produced poor results on our test site. The results for these approaches are attached in the figures attached below.
Fire Mapping: The fire mapping pipeline takes into input the three major upstream subsystems, ie., fire segmentation, fire localization (thermal depth estimation), and VIO, and fuses them to give the locations of the hotspots in the world frame. This is done by first performing clustering of the hotspots in the image plane (using the segmented mask), then projecting these hotspots in the camera frame (3D) using the depth image, and ultimately in the world frame (using VIO).

Implementation [Deployment Phase]

Under this section we briefly explain the final working pipeline that we went ahead with for our SVD. The high-level steps to calculate the coordinate of the hotspots and append them in a map were as follows

The feed from left and right thermal camera image are first rectified using cv2’s stereoRectification funciton using the camera’s distortion coefficients.
The rectified image is then converted to a temperature based binary image. This is explained in the segmentation technique above.
The mask generated from the temperature mapping is used on the rectified feeds to isolate out the hotspots and their surrounding area.
On these areas, we detect ORB features and compute key-points and descriptors for the same.
These features are then matched using a Brute-Force matcher and then filtered out using epipolar constraint.

Only those matched features are selected, which are closer to the epipolar line of their corresponding match in the other frame.
Using these good matches, we calculate disparity, and eventually depth using the baseline information.
Using proper transformations now we can calculate the 3D coordinate of the hotspot in world frame.

Fire Localization (Global Mapping)

Our global mapping subsystem handles the temporal side of the fire perception subsystem. The fire perception module provides raw measurements of hotspot locations relative to the world frame. When the fire perception subsystem publishes a new set of measurements, this mapping subsystem is executed as shown in the flowchart above.

Filtering: Once we receive a hotspot measurement, we first need to filter out the good readings, because most readings are quite noisy. The following are the 2 main filters we employ. Figure below shows the measurements from 2 successful flights (L) and the measurements after filtering (R). The red spots show the ground truth hotspot locations.

Adding New Hotspot: We then perform nearest neighbor clustering for each hotspot to determine whether it is a measurement for a new or existing one. A new hotspot is created if it exceeds distances from all previously seen hotspots.

Updating Existing Hotspot: If we instead find any measurement corresponding to an existing hotspot, we add it to the existing nearest-neighbor map, and use a mean search to find the updated position of its parent hotspot.

State Estimation

Implementation

Multi-Spectral Odometry: is a package we used from Airlab which uses a set of cameras (RGB, Thermal or a combination). The algorithm detects features in the image captured by the cameras, tracks the motion of the features as the robot platform moves and estimates the state using this information. A brief description can be seen in the following slide images:

Frame Transformations:

The MSO odometry data is estimated concerning the primary camera optical frame. For use of this data for autonomous flight with PX4, the estimates need to be transformed to a body-fixed FLU frame (centred at the IMU of the FCU). We incorporated the camera-IMU extrinsic data from Kalibr to perform this transformation.

Indoor hand-held Test: To test the drift and RMSE of the multi-spectral odometry and the IMU propagated odometry, we perform some indoor tests by dragging the Phoenix on a cart around the lab to make a path of more than 100 meters.

The 100 meter path shows a final drift of less than 1% ( of 0.912m with path length 116.562m ). Note this is using the upward looking camera in the left-front. In the field we would be using the downward facing camera in the front with appropriate mask to hide the landing gear and the sky that shows up around the edge of the fisheye feed due it’s more than 180 degree fov.

Outdoor Flight Test (Hover): (Location: Nardo)

We perform a hover test with the following criteria:

No pilot input is given to the drone
Less than 0.1 meter drift during the hover period
Hover period should be of more than 5 seconds

Video Link to hover flight footage at Nardo: https://drive.google.com/file/d/1F5KQ4VgDD9Ojskx4JA-1q7zFz8-oaJRF/view?usp=drive_link

Outdoor Flight Test (Loop): (Location: Nardo)

Loop flight footage: https://drive.google.com/file/d/115YwKfgc7a4GouLOdBDppV8XkNJJ0_i0/view?usp=drive_link

The drone was flown in a loop to evaluate the MSO drift metric in the field when compared to the GPS data. The resulting drift was of around 1.6m for a path length of 105.47m. This satisfies the performance metric that was set ( < 4% drift for a path length of 100m ).

Failsafe Policy for MSO:

As VIO by nature is unreliable in HDR and feature-sparse settings, we always need a failure detection logic to trigger appropriate failover. Some systems use the estimated confidence/covariance to make this transition. But as our VIO system does not offer such quality metrics, we rely on simple but effective data sanity checks incorporating raw IMU data and kino-dynamic limits of the system.

Having such a failsafe helps us prevent catastrophic loss of the system and/or lives.