Performance

As shown in above figure, UAV-AGV collaborative firefighting robots were demonstrated at NSH-B level where an abstract building like structure was set up using tent and polystyrene sheets. Fires were represented using hot water bags. The window and door like opening were also created for showcasing the system's ability to autonomously enter the building. Performance evaluation of Fall Validation Demonstration(FVD) is done in the following three categories: Functional, Qualitative and Quantitative

Functional Evaluation:

Given the start signal, AGV was able to drive towards the building. AGV was able to perform path-planning and reach inside the building and the UAV was able to detect window and fly through the window satisfying F.R. 1, F.R. 2 and F.R. 5
After receiving the second fire location from the UAV and AGV were able to go near that fire demonstrating the AGV's and UAV's ability to localize fire with respect common map frame and satisfying F.R. 4 and F.R. 8
The UAV and the AGV performed the mission without collision with any obstacle such as walls and window demonstrating its ability to avoid obstacle and satisfying F.R. 6
The UAV and the AGV were able to do deploy water at multiple fires demonstrating the ability to detect fire and satisfying F.R. 7 and F.R. 9
The UAV and the AGV shared the fire locations and the AGV deployed additional extinguishing material at the fire found by the UAV demonstrating their ability to collaborate through communication and satisfying F.R. 10
The UAV and the AGV shared the fire locations and the AGV deployed additional extinguishing material at the fire found by the UAV demonstrating their ability to collaborate through communication and satisfying F.R. 10

Qualitative Performance Evaluation

Some of the performance requirements from the coverage of the map to the minimum distance from the obstacle and fire are hard to measure during the FVD. These performance requirements are validated qualitatively or through subsystem testing.

For all the demonstrations during FVD and FVD Encore, the system maintained a reasonable distance with obstacles such as walls and windows to avoid the collision.
Apart from slight offset during one of the run, both the system deployed water quite accurately at the fire as shown in figure below}. In one of the run, because of hot water bags turning cold, the UAV didn't detect the fire and hence didn't deploy any water.

Quantitative Performance Evaluation

Apart from the one particular run where hot water had cooled down, UAV-AGV successfully detected all the 3 fires (100%) placed inside the "building" at various distances up to 2 m away satisfying M.P. 6
The UAV successfully carried 750 g extinguishing material and the AGV successfully carried 1 kg extinguishing material satisfying M.P. 8
While we planned to validate M.P. 9 by collecting the water received at the fire using plastic bins (at least 40%), the collection of all the water from the fire was very hard. Hence, we decided to simply measure the amount of water deployed from the UAV. The UAV and the AGV deployed close to 100% of the extinguishing material they carried and the majority of which was sprayed directly on the fires.
UAV-AGV were able to finish the joint collaborative mission in 2 minutes and 41 seconds satisfying the time aspect of M.P. 2

UAV Performance for SVD

We performed three experiments for the spring validation demonstration. In the first experiment, UAV takes off from the base station with 1.5 kg payload and then autonomously follows the predefined trajectory of four waypoints in a GPS-denied environment. As shown in below figure, UAV follows all the required waypoint within the error limits of ± 50 cm. Similarly, as shown in the below figure; there is a maximum error of ± 50 cm in the x and y-direction & ± 20 cm in the z-direction. UAV detects fire whenever its at max 150 cm away from the simulated fire.

After reaching the third waypoint, UAV demonstrates its fire extinguishing capability by pointing a laser towards the simulated fire (Hot water bag). As shown in the below figure, UAV was oscillating around the yaw axis while performing this task during SVD. We significantly reduced the yaw error in SVD-encore and maximum error was around 45 cm and the mean error was 39cm through the extinguisher task.

UGV Performance for SVD

In the second experiment, AGV drives off from its base station and follows a prede- fined trajectory in GPS-denied environment with a maximum tracking error of 50 cm in x & y-direction and 20 cm in the z-direction. AGV is able to detect fire from maximum 150cm and it is able to track the fire with an error of around 15cm. AGV detects all the obstacles within 75cm of range and stops immediately.

In the third experiment, we have shown a video demonstration of UR5 arm video tracking the fire (hot water massage bag). It was able to track the fire with an error of ± 15cm. So, we were able to meet all our SVD performance metrics for both UAV & AGV.

Overall system performance mapped to the system requirements

Requirement ID	Requirement Description	Corresponding Subsystem
M.P.1	AGV successfully parks in within 1 m radius of the base station center	Navigation Control
M.P.7	AGV successfully detects fire from a maximum 1.5 m distance	Fire detection and Localization
M.P.5	AGV successfully stops as soon as it detects obstacle 0.75 m away and does not crash into it	Navigation Control
M.P.8	AGV points a laser pointer within the water bag of size 7.5 inch x 9.5 inch	Fire extinguisher
M.P.9	UAV successfully lifts 1.5 KG payload	Navigation Control
M.P.7	UAV successfully detects the hot water bag from a maximum of 1.5 m distance.	Fire detection and Localization
M.P.9	UAV points a laser pointer within the wa- ter bag of size 7.5 inch x 9.5 inch	Fire extinguisher

The above table shows the list of the system requirements and corresponding subsystems and system elements emphasized during the spring semester development. Most of the re- quirements in this semester were focused towards Navigation Control, Fire detection and Localization, Fire extinguishing subsystem. M.P.1 & M.P.4 verified our Navigation Control subsystem where a majority of the work was done this semester. This func- tionality enabled us to test and validate our M.P.7 (Fire detection and Localization sub- system). Further, we validated our Fire extinguisher subsystem through M.P.8 & M.P.9. Since our Fire detection and localization & Fire extinguisher subsystem code was made modular, the same code worked on both UAV and AGV without many changes. Sur- prisingly the SVD satisfies more than 50% of our mandatory performance requirements and thus we are optimistic in touching upon some desirable requirements if time permits in the next semester.