{"id":32,"date":"2019-01-17T01:29:10","date_gmt":"2019-01-17T01:29:10","guid":{"rendered":"http:\/\/mrsdprojects.ri.cmu.edu\/2018teamd\/?page_id=32"},"modified":"2019-12-10T05:42:02","modified_gmt":"2019-12-10T10:42:02","slug":"system-performance","status":"publish","type":"page","link":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teamd\/system-performance\/","title":{"rendered":"System Performance"},"content":{"rendered":"

Mandatory Performance Requirements<\/strong><\/h3>\n

M.P.1. Explore, scan and create a 2D map for 90% of the reachable area in a room.<\/span><\/b><\/p>\n

M.P.2. (Scaled)\u00a0Clean up a 10m\u00b2 room with 5 tennis-ball-sized objects within 10 minutes.<\/span><\/b><\/p>\n

M.P.3. Navigate to a designated reachable location in a room with pose error < 10%.<\/span><\/b><\/p>\n

M.P.4.\u00a0<\/b>Go over carpets and rugs with thickness less than 12mm.<\/span><\/b><\/p>\n

M.P.5. Detect and avoid 75% of the obstacles with a clearing distance of 20cm.<\/span><\/b><\/p>\n

M.P.6. Classify all tennis ball-sized objects with classification error < 20%.<\/span><\/b><\/p>\n

M.P.7. Pick up and collect each classified object within 5 attempts.<\/span><\/b><\/p>\n

M.P.8. Pick up at least 80% of the classified objects in the room.<\/span><\/b><\/p>\n

M.P.9. Drop the clutter in a designated container with a success rate > 90%.<\/span><\/b><\/p>\n

Fall Validation Demonstration (December 2019)<\/strong><\/h3>\n

General Description<\/strong><\/h4>\n

The high-level goal for our Fall Validation Demonstration (FVD) is to pick up 4 out of 5 toys (Figure 18) in a room in under 10 minutes. It was able to pick up all five toys in only 9 minutes. Examples of the toys can be seen in the figure below.\u00a0<\/span><\/p>\n

\"\"<\/p>\n

<\/b><\/p>\n

With the extra time, we tested Cubi again. During the second test, we asked the evaluators to try to break our system by placing smaller toys, plastic cups, tape and tape measures.\u00a0 Our system would always detect the toys. However, 5% of the time we would miss them when approaching them. This was because our vision system is not robust when detecting toys in motion. In terms of picking up objects, we were able to pick up all objects over 95% of the time if we aligned properly. We had difficulty when trying to pick up a plastic cup as the fingers on the gripper were not long enough to push it in. In this case, our system was able to detect that it had not picked up an object and that the plastic cup was stuck in the fingers.\u00a0<\/span><\/p>\n

Test Setup<\/strong><\/h4>\n

1. Before testing, Team CuBi will have already created a 2D map of the walls of the room.\u00a0<\/span><\/p>\n

2. Third-party places 5 toys at least 30 cm away from obstacles (including walls) at any location they want.\u00a0<\/span><\/p>\n

3. Team CuBi will place a box at the starting location. This will designate where the toys will be placed by the end of the <\/span>10 minutes<\/span><\/i>. Toys will be placed in a box.<\/span><\/p>\n

4. CuBi is placed in the designated starting position.<\/span><\/p>\n

Test Procedure<\/strong><\/h4>\n

 <\/p>\n\n\n\n\n\n\n\n\n\n\n\n
#<\/b><\/td>\nDescription<\/b><\/td>\nPerformance Measures<\/b><\/td>\n<\/tr>\n
1<\/span><\/td>\nLaunch CuBi.<\/span><\/td>\n<\/td>\n<\/tr>\n
2<\/span><\/td>\nCuBi starts to explore the room and perform SLAM to add static obstacles to 2D map of the room.<\/span><\/td>\n90% of the reachable area should be mapped by the robot<\/span><\/td>\n<\/tr>\n
3<\/span><\/td>\nCuBi performs local planning to traverse the room while avoiding the obstacles it sees.<\/span><\/td>\nAvoid 75% of the obstacles<\/span><\/td>\n<\/tr>\n
4<\/span><\/td>\nWhenever CuBi sees a toy, it uses its manipulator to pick it up.\u00a0<\/span><\/td>\nManipulator should pick up toy within 5 attempts<\/span><\/td>\n<\/tr>\n
5<\/span><\/td>\nCuBi should be able to drop the clutter at the designated position marked with AprilTag accurately.<\/span><\/td>\nThe success rate of dropping should be more than 90%.<\/span><\/td>\n<\/tr>\n
6<\/span><\/td>\nCuBi will reset its odometry every time it drops off a toy.\u00a0<\/span><\/td>\nWill localize indoors with accumulated error < 10% per 20 minutes of operation.<\/span><\/td>\n<\/tr>\n
7<\/span><\/td>\nCuBi should be able to pick up most of the toys off the ground.<\/span><\/td>\nAt least 80% of the toys should be picked up by CuBi\u00a0<\/span><\/td>\n<\/tr>\n
8<\/span><\/td>\nCuBi should be able to clean up 10m\u00b2 area in a reasonable time.<\/span><\/td>\nIt should pick 4 out of 5 toys in less than 10 minutes<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Evaluation Result<\/b><\/h4>\n

We have successfully achieved all our mandatory system performance requirements in the Fall Validation Demonstration Encore.<\/span><\/h4>\n

Check out the video of CuBi in action here<\/a>!<\/h4>\n
(Below are the performance evaluation before May 2019)<\/span><\/h6>\n

Spring Semester Targeted System Requirements<\/strong><\/h3>\n

M.P.2. (Scaled)\u00a0<\/b>Clean up a <\/span>~4m\u00b2 area<\/span> with <\/span>5 tennis-ball-sized objects<\/span> within <\/span>10 minutes<\/span>
\n<\/span>
\n<\/span>M.P.3.<\/b> Navigate to a designated reachable location in a room with pose error < 10%<\/span>
\n<\/span>
\n<\/span>M.P.6.<\/b> Classify all tennis ball-sized objects with classification error < 20%<\/span>
\n<\/span>
\n<\/span>M.P.7.<\/b> Pick up and collect each classified object within 5 attempts<\/span>
\n<\/span>
\n<\/span>M.P.8.<\/b> Pick up at least 80% of the classified objects in the area<\/span>
\n<\/span>
\n<\/span>M.P.10.<\/b> Drop the clutter in a designated container <\/span>at a predetermined location with success rate > 90%<\/span><\/p>\n

We have achieved all the targeted system requirements in the Spring Validation Demonstration. Moreover, we also beat most of the metrics stated in the requirements by 20% – 50%.<\/p>\n

Spring Validation Demonstration<\/strong><\/h3>\n

\"\"<\/h4>\n

Test Procedure<\/b><\/h4>\n

1. Randomly place 5 toys inside the designated area<\/span><\/p>\n

2. Turn on Cubi.<\/span><\/p>\n

3. Search for a toy and move towards it.<\/span><\/p>\n

4. Pick up the toy.<\/span><\/p>\n

5. Return to the start position.<\/span><\/p>\n

6. Drop the toy to a designated drop-off location (into a box). <\/span><\/p>\n

7. Repeat steps 2 to 5 until all toys are picked up.<\/span><\/p>\n

8. If Cubi fails, we place Cubi at the start location and turn it on again. <\/span><\/p>\n

Verification Criteria<\/b><\/h4>\n

Overall Goal<\/strong><\/p>\n

Pick up 3 toys and drop them to a designated location <\/span>within 25 minutes.<\/span><\/p>\n

Secondary Goals<\/strong><\/p>\n

    \n
  1. \n
      \n
    1. Detect a toy and calculate its pose relative to Cubi<\/span><\/li>\n
    2. Approach a toy within 2 cm<\/span><\/li>\n
    3. Pick up one toy on the tray<\/span><\/li>\n
    4. Lift the tray (with a toy inside) up from the ground.<\/span><\/li>\n
    5. Return to the start position<\/span><\/li>\n
    6. Drop a toy in the box<\/span><\/li>\n<\/ol>\n<\/li>\n<\/ol>\n

      Evaluation Result<\/b><\/h4>\n

      Overall Goal<\/strong><\/p>\n

      Picked up <\/span>all 5 toys<\/strong> and dropped them into a box <\/span>within 10 minutes<\/strong>.<\/span><\/p>\n

      Secondary Goals<\/strong><\/p>\n

      All the secondary goals are achieved.<\/p>\n

       <\/p>\n","protected":false},"excerpt":{"rendered":"

      Mandatory Performance Requirements M.P.1. Explore, scan and create a 2D map for 90% of the reachable area in a room. M.P.2. (Scaled)\u00a0Clean up a 10m\u00b2 room with 5 tennis-ball-sized objects within 10 minutes. M.P.3. Navigate to a designated reachable location in a room with pose error < 10%. M.P.4.\u00a0Go over carpets and rugs with thickness … Continue reading System Performance<\/span><\/a><\/p>\n","protected":false},"author":151,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-32","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teamd\/wp-json\/wp\/v2\/pages\/32","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teamd\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teamd\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teamd\/wp-json\/wp\/v2\/users\/151"}],"replies":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teamd\/wp-json\/wp\/v2\/comments?post=32"}],"version-history":[{"count":9,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teamd\/wp-json\/wp\/v2\/pages\/32\/revisions"}],"predecessor-version":[{"id":498,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teamd\/wp-json\/wp\/v2\/pages\/32\/revisions\/498"}],"wp:attachment":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teamd\/wp-json\/wp\/v2\/media?parent=32"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}