| Functionality | System Requirement | Achieved System Performance |
| 2D Fire Map | >= 80% detection accuracy | 88% |
| 2D Fire Map | >= 80% association accuracy | 100% |
| Autonomous Flight | >= 667m2 per minute | 2000 m2 per minute (can be adjusted according to user preference) |
| 2.5D Terrain Map | <= 5m height accuracy | Average Accuracy: 1.067m Maximum Error: 4.164m |
Our system’s achieved performance against the benchmarks derived from our Mandatory Performance Requirements are summarized in the table above.
Detection accuracy: This is a measure of how well our system is able to label a ground truth fire occupancy grid cell correctly. This evaluated by first generating groud truth labels for the cells by using RTK GPS measurements of our control hotspot locations and tagging all cells within a 5m radius as a fire. The mapping output of our system generating after surveying the landscape is then compared against this to evaluate our accuracy.
Association Accruacy: This is a measure how many individual isolated pockets of fire have been detected, with the expectation that at least 80% of these get detected. This is evaluated by keeping a count of how many ground truth fire locations have lies within a 5m radius of a cell that has been detected as a fire.
Flight Time: Our flight time for a given area is determined by our requirement of having to map at a rat e of at least 667m2/ min
Height Accuracy: The real-time 2.5D created by the onboard system during flight processes lidar points to get the max height of each grid cell with a resolution of 0.5m. To test the the accuracy of our 2.5D terrain map we collected a Faro scanner point clouds of the Mall at CMU. This point cloud was processed in the same manner as the onboard lidar data. The RTK position of drone starting position and RTK position of the Faro scan are used to align maps and the corresponding bins between maps are compared. Absolute difference in height between bins are tracked and averaged to get maximum and average height error respectively.