System Performance

FVD Performance Evaluation

Test 1: Functionality Demonstration

Criteria Performance
UAV and UGV collecting samples at different positions Fulfilled
UAV taking measurement above obstacles Fulfilled
UGV avoiding obstacles Fulfilled

Test 2: Performance/Accuracy Demonstration

Criteria Requirement Performance
RMS Error of Temperature < 2 ºC 1.5509 ºC

The following image shows the predicted temperature distribution model of the test field (left) and the ground truth temperature distribution model (right). The RMSE of the predicted temperature model is 1.5509ºC. Ground truth model was obtained by manually taking uniform samples from the area and estimated by interpolating the data. This interpolated model is only used for for visualization. The prediction error is calculated by directly using those uniformly collected samples. Please also refer to the FVD video under the media section for more details on the experiment.

FVD Performance Conclusions

Strength:

  • Stable and robust system architecture:
    • The robot agents share the same code base and obey the standard communication rules between the master computer.
    • The temperature modeling algorithm and sampling algorithm also do not depend on the number of robot agents in the system.
    • easy scale up to robot fleets and perform informative sampling in a broader scope.

Weakness:

  • Operation Efficiency:
    • The current operation speed leads to a more than 30-minute time required to wait for the temperature model to converge.
    • Tuning operation speed can make the system more efficient, and it may also require a more robust motion of the Jackal.
  • Communication Coverage:
    • Currently using four wifi boosters to establish a communication coverage over the 10m x 10m x 5m test field.
    • Naively adding more wifi boosters to the field is not a feasible method to deploy the system in a large and sophisticated open area.
    • Need a more robust and scalable communication method to make our use case come true.

SVD Performance Evaluation

Test 1: Master Computer Subsystem Validation Experiment

Criteria Requirement Performance
RMS Error of Temperature < 1 ºC 0.35 ºC
Iterations before temperature model converges (average variance less than 1 ºC) < 50 Loops 30 +/- 10 Loops

Test 2 & 3: UGV/UAV Subsystem Validation Experiment

Criteria Requirement Performance of UAV Performance of UGV
Mean Location Error < 2 m 0.38m 0.1m
Mean Temperature Error < 2 ºC 1.2 ºC 0.4 ºC

SVD Performance Conclusions

Strength:

  • Great navigation accuracy of both UAV and UGV.
  • Great convergence and generality of informative sampling algorithm.
  • Complete simulation system for unit tests.

Weakness:

  • Current temperature sensor convergence rate could be as low as 3 ~ 5 minutes for a 5 ℃ temperature gap.
  • Communication latency, including latency and occasional freeze.

Future Improvements:

  • Improve temperature measurement efficiency.
  • Refinement of heat source.