FVD Performance Evaluation
Test 1: Functionality Demonstration
| Criteria | Performance |
| UAV and UGV collecting samples at different positions | Fulfilled |
| UAV taking measurement above obstacles | Fulfilled |
| UGV avoiding obstacles | Fulfilled |
Test 2: Performance/Accuracy Demonstration
| Criteria | Requirement | Performance |
| RMS Error of Temperature | < 2 ºC | 1.5509 ºC |
The following image shows the predicted temperature distribution model of the test field (left) and the ground truth temperature distribution model (right). The RMSE of the predicted temperature model is 1.5509ºC. Ground truth model was obtained by manually taking uniform samples from the area and estimated by interpolating the data. This interpolated model is only used for for visualization. The prediction error is calculated by directly using those uniformly collected samples. Please also refer to the FVD video under the media section for more details on the experiment.
FVD Performance Conclusions
Strength:
- Stable and robust system architecture:
- The robot agents share the same code base and obey the standard communication rules between the master computer.
- The temperature modeling algorithm and sampling algorithm also do not depend on the number of robot agents in the system.
- easy scale up to robot fleets and perform informative sampling in a broader scope.
Weakness:
- Operation Efficiency:
- The current operation speed leads to a more than 30-minute time required to wait for the temperature model to converge.
- Tuning operation speed can make the system more efficient, and it may also require a more robust motion of the Jackal.
- Communication Coverage:
- Currently using four wifi boosters to establish a communication coverage over the 10m x 10m x 5m test field.
- Naively adding more wifi boosters to the field is not a feasible method to deploy the system in a large and sophisticated open area.
- Need a more robust and scalable communication method to make our use case come true.
SVD Performance Evaluation
Test 1: Master Computer Subsystem Validation Experiment
| Criteria | Requirement | Performance |
| RMS Error of Temperature | < 1 ºC | 0.35 ºC |
| Iterations before temperature model converges (average variance less than 1 ºC) | < 50 Loops | 30 +/- 10 Loops |
Test 2 & 3: UGV/UAV Subsystem Validation Experiment
| Criteria | Requirement | Performance of UAV | Performance of UGV |
| Mean Location Error | < 2 m | 0.38m | 0.1m |
| Mean Temperature Error | < 2 ºC | 1.2 ºC | 0.4 ºC |
SVD Performance Conclusions
Strength:
- Great navigation accuracy of both UAV and UGV.
- Great convergence and generality of informative sampling algorithm.
- Complete simulation system for unit tests.
Weakness:
- Current temperature sensor convergence rate could be as low as 3 ~ 5 minutes for a 5 ℃ temperature gap.
- Communication latency, including latency and occasional freeze.
Future Improvements:
- Improve temperature measurement efficiency.
- Refinement of heat source.
