The team had a few major components that required testing throughout the semester.
Fire Segmentation
For the perception subsystem, the fire segmentation capabilities were isolated and tested on various objects. From this process, we found that there were some instances of false positives in which the fire segmentation subsystem mistakenly identified hot car roofs as fires. We believe this error is mainly due to the fact that the threshold for the fire segmentation pipeline is currently only set to 50°C; the reason that the value is set so low is because we are using small handheld burners at the moment, and at high operating heights, the thermal camera averages out the temperature of the stoves with the colder ground. We are hopeful that in the upcoming semester we will be able to increase the threshold since we will be trying to test our system on actual open fires which will be hotter and cover a larger area.
Mapping Time Profiles
For the mapping subsystem we had conducted timing profiles in order to test how fast each component of the mapping pipeline is. The conclusion of these testing results showed that as long as we build our files in release mode, our system should process fast enough to map each frame of the thermal camera in real-time.
Real-Time Telemetry
For the telemetry subsystem, we conducted a variety of radio tests. We ensured that we could send data packets at a range greater than 200 meters in order to meet performance requirement M.P.3. We also ensured that we could send full map updates generated from an operating height of 15 meters at least once every ten seconds in order to meet performance requirement M.P.2. While the range and bandwidth of the system are sufficient enough for our current system to operate in real-time and meet performance requirements M.P.2 and M.P.3, bandwidth limitations may restrict the type and amount of data we can send in the future.
Custom PCB (Onboard Temperature Monitoring)
During the course of the semester, we were also required to design and integrate a custom PCB board into our system. The PCB board that was designed is used to monitor the ambient temperature of the drone. To test the functionality of this PCB, we used an IR thermometer in order to measure the temperature of the IC and compare this value with the output reading. This test showed that the PCB is able to read the ambient air temperature fairly reliably with a maximum error of about ±2°C. This is sufficient for the use case of the PCB which is to ensure the drone does not overheat during operation over intense fires.
Mapping Accuracy
The spring validation demonstration itself served as a test for our mapping pipeline as well as our overall integrated system. During the demonstration we measured the ground truth values of the hotspots with an RTK test board and we validated the results of our flight using a custom validation script. This validation script outputs the accuracies of our system as described by our system requirements. Over multiple runs, we were able to demonstrate that our mapping accuracy is consistently above our performance requirements.