Performance

Fall Validation Demonstration

The Fall Validation Demonstration was conducted in two parts, in simulation, and on hardware. The following sections detail the tests performed and results obtained for both demos.

Simulation

In the simulation, we showcased the chassis’ ability to plan a path to the desired pod, identify it, and dock with it. It would then drop the pod off at a predefined undocking location. The validation criteria and results for the demo were:

1. The system will continuously log the system state and health. State transitions occur only when the last action has been completed. The logs will present system state, health, and results of actions in an easily readable manner.

The Behavioural State Machine node and the Health Monitoring System operated continuously throughout the length of the simulation. The system state was logged by the State Machine. It outputted the current time, current operation mode (pickup, drop-off, idle), current state, the action performed and its output, and the state the system will transition into as a result of the action taken. The Health Monitoring System also outputted the results of sanity and persistence checks for all nodes and sensors.

    2.  (Un)Docking will be done within 120s.

This criterion satisfies M.P.2. 4 processes are subsumed within this requirement: approach navigation to either pick up or drop-off the pod, final pose verification when the approach is complete, retrace if the pose achieved is outside error bounds, and the physical act of docking and lifting up the pod or undocking and placing it on the ground. The chassis executed all the procedures well within 120s in all scenarios including those in which the chassis had to retrace its path to the start of the Payload Handling Zone (PHZ) except when approach navigation fails due to a large difference in the initial pose of the chassis from the ideal initial pose at the start of the PHZ. In order to induce this failure, we had to manually shift the chassis in the simulation from the pose it had achieved at the end of the point-to-point (P2P) navigation. Our P2P algorithm ensures that the probability of occurrence of such cases is low.

    3.  The simulation should have a success rate of 85%.

The success criterion subsumes M.P.1 (chassis will identify pod when it is within 2m of the PHZ), M.P.3 (chassis will achieve docking pose with an error margin of ±5 cm in position and ±5.54 degrees in orientation), and M.P.8 (chassis will identify PHZ 90% of the time). During testing, the P2P navigation was always successful, and the chassis was always able to detect it’s proximity to the PHZ. The chassis was able to achieve a docking pose within the error bounds imposed by M.P.3 either on the first try or after retracing in 90% of the cases. The system was also able to detect, report, and diagnose system failures within 2000ms (M.P.5). It also maintained a minimum distance of 30cm at all times from objects (M.P.7) and determined their pose with an accuracy of 5cm (M.P.6). Overall, the entire process was successful 7 out of 8 times (87.5% success rate).

Table 1 below lists the unit tests that were performed to validate each of the mandatory performance requirements along with the success rate of the test.  These tests were performed in addition to the Fall Validation demonstration to ensure the successful operation of the entire system.

Hardware

We also validated some of the algorithms we had developed with real-world data using a makeshift mobile platform with sensors mounted on top of it. The metrics that were verified and the results are described below.

1. Predicted pod position (x,y) for relative localization is within 20 cm of the ground-truth, and the orientation (yaw) is within ±11.80 degrees of the ground-truth. The same margins hold for the predicted docking pose (x,y, yaw) error and actual error.

For this test, we first placed the mobile platform in front of the pod where it was able to view the AprilTags mounted on the pod. The system was able to predict the pod’s detected location to within the accuracy bounds mentioned above even when the platform was rotated and translated in different directions. The platform was then moved underneath the pod. It was able to predict the docking pose within the bounds as well.

    2.  Object locations are detected with an accuracy of ± 5 cm (M.P.6).

This requirement was met. We also demonstrated the pedestrian tracking ability of our system. Our system was able to identify moving pedestrians and differentiate them from static obstacles. It was also able to reassign them the same ID when they reentered the camera’s field of view.

Table 1: FVD Performance Evaluation

Spring Validation Demonstration

Targeted Requirements

For the Spring, we targeted the requirements that relate to the process of navigating to the pod and then aligning with it. Detailed requirements can be found here.

Table 1: Targetted Requirements

Performance

The following table lists the tests we had planned for the Spring Validation Demonstration, their description, and the results that we obtained in our preliminary tests as well as in both the validation demonstrations.

The video can be found here.

Table 2: SVD Performance Evaluation

Detailed Tests Performed

This section will give the detailed tests performed and which subsystems and requirements were validated.

1. Navigation

Subsystems Validated: Simulation and Navigation Subsystem (Localization, Detection, and Point to Point Navigation)

Table 3: Navigation Subsystem Tests

2. Safety

Subsystems Validated: Safety

Table 4: Safety Subsystem Tests

3. Align 

Subsystems Validated: Navigation (Approach Navigation) and Docking

Table 5: Align Navigation Tests

4. Retracing

Subsystems Validated: Docking

Table 6: Retracing Tests