Fall Validation Performance Evaluation
Test 1: Realistic Simulator

Test 2: Full Mission Validation
The full mission video can be accessed at https://youtu.be/u0G5B1FtIYM




Spring Validation Performance Evaluation
Test 1: Characterization of Detection
This test validates the detection pipeline of the system. 30 pictures of the rover were captured using the catadioptric camera. The detection algorithm was then executed on these images. The algorithm returned a bounding box on the rover and also gave as an output the pixel value of the rover centroid. This was then compared to the ground truth. The ground truth value was found out by manually clicking on the centroid of the rover in the same image. If these two centroids were within a margin of 40 pixels, the detection was marked correct. This threshold was configured taking into account the bearing error we may obtain if the detection is within 40 pixels of the true value for a 2000 X 2000 pixels image. We had stated precision of 80% in our requirements and we were able to achieve a precision of 92%.
Figure: (a) Ground truth centroid in red (b) Centroid obtained via detection algorithm in blue
Test 2: Characterization of Range, Bearing and Communication
This test was divided into three sub-tasks:
- The first corresponds to the validation of the range and the bearing measurement. The range measured by the UWB Decawave sensor gave a mean error of 14 cm against the stated requirement of 30cm. This was validated by driving the rover and then measuring the ground truth using a 12m measuring tape. The mean bearing error was stated at 4°. To validate the bearing measurement the x and y values given by the RTK GPS were taken and then the ground truth bearing was computed by applying trigonometry on the same. The following requirement was not validated completely due to unreliable and fluctuating estimates.
- The next requirement corresponds to the successful capture of sensor data after 15 seconds. The stated requirement was 70% of the time. This requirement was validated by using the real-time visualization dashboard which depicted when the sensor measurements were recorded. We achieved a successful sensor capture accuracy of 100%.
- The third task pertains to the validation of the communication range. We achieved a communication range of 35.3 m between the two rovers against the stated 20m range. This was validated by pinging one rover from the other. If the second rover was able to receive the ping this implies the communication is consistent.
Test 3: Characterization of Range, Bearing and Communication
This test was divided into three sub-tasks
- The first corresponds to the validation of the RTK GPS. This was validated by using measuring tape as the ground truth. We promised the values from the RTK GPS to be within ± 6cm of the true value. We were able to validate this deviation to a tighter bound of ± 2.5cm
- The next requirement corresponds to the successful completion of co-localization. We stated that we would be able to achieve an updated path of the rovers within an error of 5% of the distance traveled.
- The third task pertains to the validation of localization using odometry/EKF. This was expected to be within 20% of the ground truth value (measured from the RTK) for every 100m traveled. This requirement was not validated.