Fall Validation Demonstration Results
We performed two sets of full system tests for our Fall Validation Demos; one in simulation, and one on the Husky platform. The desired requirements for both tests are listed below.
- Final translational error < 75 cm
- Final rotational error < 10 deg
- Total runtime < 2 x tele-op runtime
a) Simulation Results:
To fully validate system robustness, we created four Gazebo worlds. These four worlds cover all types of scenarios we expect our system to be able to handle for parking lot traversal (making right/left turns around the bend) and parking (parking on the right/left side of the road). For each world, we performed three autonomous runs (one on each of our laptops simultaneously) and compared their results with one manually teleoperated run (on Nikita’s laptop). The two tables below show our final results for the world in which the Husky makes two right turns around the bend and parks in a parking spot to its right side.
| RRR World | Nikita (Tele-operator) | Subbu (Machine 1) | Eva (Machine 2) | Kevin (Machine 3) | Average |
|---|---|---|---|---|---|
| Time (s) | 77 | 86 | 105 | 122 | 104.33 |
| Parking Translational error (m) | N/A | 50.03 | 60.20 | 78.95 | 63.06 |
| Parking Rotational error (deg) | N/A | 20.3 | 20.70 | 16.27 | 19.09 |
| Success Criteria | Results | % Success Rate | Avg. deviation from desired value |
|---|---|---|---|
| Time < 154s | PASS | 100% | 0 |
| Translational error < 75% | PASS | 66.66% | 0 |
| Rotational error < 10 deg | FAIL | 0% | 9.09 degrees |
The tables below show our final results for the world in which the Husky takes two left turns around the bend and parks in a parking spot to its left side.
| LLL World | Nikita (Manual) | Subbu (Machine 1) | Eva (Machine 2) | Kevin (Machine 3) | Average |
|---|---|---|---|---|---|
| Time(s) | 94 | 87 | 85 | 94 | 88.67 |
| Parking translational error (cm) | N/A | 34.45 | 13.85 | 20.19 | 22.83 |
| Parking rotational error (deg) | N/A | 0.24 | 11.85 | 0.84 | 4.31 |
| Success Criteria | Results | % Success Rate | Avg. deviation from desired value |
|---|---|---|---|
| Time < 188s | PASS | 100% | 0 |
| Translational error < 75 cms | PASS | 100% | 0 |
| Rotational error < 10 degrees | PASS | `66.66% | 1.85 degrees |
Inference: All 6 runs in simulation passed the timing criteria and 2 out of 6 runs were fully successful in all aspects. There were 5 out of 6 successful runs with respect to translational errors, and 2 out of 6 successful runs with respect to orientational errors. Overall, the unsuccessful runs were only off by an average of 3.95 cm in translation and 7.28 degrees in orientation. Here is a link to one of our test runs in simulation.
b) Hardware results:
We performed tests to validate full system functionality for two scenarios–two right turns and parking on the right side, and two right turns and parking on the left side. At the end of our demonstration, we asked the instructors to select a random parking spot, and the Husky was able to successfully park itself within the requirements. Here is a link to one of our test runs on hardware.
1. Two right turns and right-side parking –
- RESULTS: Time = 158s;
- Translation error = 66cm;
- Rotation error = 2.1 degrees
- INFERENCE: Mostly successful. During this run, the robot froze on one of the turns, we had to give it a slight “nudge” with the joystick, and then it got back on track and finished autonomous functionality.
2. Two right turns and left-side parking –
- RESULTS: Time = 115s;
- Translation error = 50cm;
- Rotation error = 8.6 degrees
- INFERENCE: Successful full run within all requirements.
3. Two right turns and right-side parking –
- RESULTS: Time = 101s;
- Translation error = 95cm;
- Rotation error = 5.8 degrees
- INFERENCE: Successful full run, but translation error above requirement by 20 cm
4. Random parking spot selection –
- RESULTS: Time = 66s;
- Translation error = 38cm;
- Rotation error = 5.1 degrees
- INFERENCE: Successful full run within all requirements
Spring Validation Demonstration Results
| Requirement | Description | Achieved |
|---|---|---|
| M.P.2 | System will create a map of the environment with a mapping error of less than 20 cm as compared to the ground truth map | 16.305 cm |
| M.P.3 | System’s map building will be robust over sharp turns, such that the SLAM pipeline will not break (lose track of correspondences) in 80% of trials where a turn is present | 100% |
| M.P.4 | System will discretely localize itself within 5 degrees in rotation and 50 cm in translation as compared to the ground truth location | 48.836 cm translation, 2.099 deg rotation |
| M.P.5 | System will detect lanes in all frames with an Intersection over Union (IOU) score of greater than 60% | 98% |
SVD Test 1 (Mapping Test)
- We placed several unit boxes with known locations in our custom Gazebo environment.
- The Husky was tele-operated around the parking lot, creating a map of the world
- We ran an evaluation script where the user selects the map points corresponding to the corners of the unit boxes.
- These map points were compared against their known ground-truth locations, and an average error was calculated across all points.
- We validated that this average error was within our requirement (M.P.2, M.P.3)
Figure 1. Mapping test setup (left) and calculated euclidean error across each unit box point (right)
SVD Test 2 (Localization Test)
- The Husky was tele-operated around the simulated parking lot
- Every 0.3 meters travelled, our script output the Husky’s calculated pose estimate and compared it against the ground truth pose
- We validated that the average pose error in both translation and rotation was within our requirement (M.P.4)
SVD Test 3 (Lane Detection Test)
- The Husky was tele-operated around the simulated parking lot
- The calculated lane segmentation was compared against the known ground-truth lane region, and the mean intersection over union (IOU) was reported
- We validated that the mean IOU across all frames was within our requirement (M.P.6)
Figure 2. Image seen from the simulated camera (left) and detected lane (right image). The IOU for this frame is shown in the top righthand corner.