Mandatory Performance Requirements
M.P.1. Explore, scan and create a 2D map for 90% of the reachable area in a room.
M.P.2. (Scaled) Clean up a 10m² room with 5 tennis-ball-sized objects within 10 minutes.
M.P.3. Navigate to a designated reachable location in a room with pose error < 10%.
M.P.4. Go over carpets and rugs with thickness less than 12mm.
M.P.5. Detect and avoid 75% of the obstacles with a clearing distance of 20cm.
M.P.6. Classify all tennis ball-sized objects with classification error < 20%.
M.P.7. Pick up and collect each classified object within 5 attempts.
M.P.8. Pick up at least 80% of the classified objects in the room.
M.P.9. Drop the clutter in a designated container with a success rate > 90%.
Fall Validation Demonstration (December 2019)
General Description
The high-level goal for our Fall Validation Demonstration (FVD) is to pick up 4 out of 5 toys (Figure 18) in a room in under 10 minutes. It was able to pick up all five toys in only 9 minutes. Examples of the toys can be seen in the figure below.
With the extra time, we tested Cubi again. During the second test, we asked the evaluators to try to break our system by placing smaller toys, plastic cups, tape and tape measures. Our system would always detect the toys. However, 5% of the time we would miss them when approaching them. This was because our vision system is not robust when detecting toys in motion. In terms of picking up objects, we were able to pick up all objects over 95% of the time if we aligned properly. We had difficulty when trying to pick up a plastic cup as the fingers on the gripper were not long enough to push it in. In this case, our system was able to detect that it had not picked up an object and that the plastic cup was stuck in the fingers.
Test Setup
1. Before testing, Team CuBi will have already created a 2D map of the walls of the room.
2. Third-party places 5 toys at least 30 cm away from obstacles (including walls) at any location they want.
3. Team CuBi will place a box at the starting location. This will designate where the toys will be placed by the end of the 10 minutes. Toys will be placed in a box.
4. CuBi is placed in the designated starting position.
Test Procedure
| # | Description | Performance Measures |
| 1 | Launch CuBi. | |
| 2 | CuBi starts to explore the room and perform SLAM to add static obstacles to 2D map of the room. | 90% of the reachable area should be mapped by the robot |
| 3 | CuBi performs local planning to traverse the room while avoiding the obstacles it sees. | Avoid 75% of the obstacles |
| 4 | Whenever CuBi sees a toy, it uses its manipulator to pick it up. | Manipulator should pick up toy within 5 attempts |
| 5 | CuBi should be able to drop the clutter at the designated position marked with AprilTag accurately. | The success rate of dropping should be more than 90%. |
| 6 | CuBi will reset its odometry every time it drops off a toy. | Will localize indoors with accumulated error < 10% per 20 minutes of operation. |
| 7 | CuBi should be able to pick up most of the toys off the ground. | At least 80% of the toys should be picked up by CuBi |
| 8 | CuBi should be able to clean up 10m² area in a reasonable time. | It should pick 4 out of 5 toys in less than 10 minutes |
Evaluation Result
We have successfully achieved all our mandatory system performance requirements in the Fall Validation Demonstration Encore.
Check out the video of CuBi in action here!
(Below are the performance evaluation before May 2019)
Spring Semester Targeted System Requirements
M.P.2. (Scaled) Clean up a ~4m² area with 5 tennis-ball-sized objects within 10 minutes
M.P.3. Navigate to a designated reachable location in a room with pose error < 10%
M.P.6. Classify all tennis ball-sized objects with classification error < 20%
M.P.7. Pick up and collect each classified object within 5 attempts
M.P.8. Pick up at least 80% of the classified objects in the area
M.P.10. Drop the clutter in a designated container at a predetermined location with success rate > 90%
We have achieved all the targeted system requirements in the Spring Validation Demonstration. Moreover, we also beat most of the metrics stated in the requirements by 20% – 50%.
Spring Validation Demonstration
Test Procedure
1. Randomly place 5 toys inside the designated area
2. Turn on Cubi.
3. Search for a toy and move towards it.
4. Pick up the toy.
5. Return to the start position.
6. Drop the toy to a designated drop-off location (into a box).
7. Repeat steps 2 to 5 until all toys are picked up.
8. If Cubi fails, we place Cubi at the start location and turn it on again.
Verification Criteria
Overall Goal
Pick up 3 toys and drop them to a designated location within 25 minutes.
Secondary Goals
-
- Detect a toy and calculate its pose relative to Cubi
- Approach a toy within 2 cm
- Pick up one toy on the tray
- Lift the tray (with a toy inside) up from the ground.
- Return to the start position
- Drop a toy in the box
Evaluation Result
Overall Goal
Picked up all 5 toys and dropped them into a box within 10 minutes.
Secondary Goals
All the secondary goals are achieved.