{"id":74,"date":"2020-01-15T04:22:53","date_gmt":"2020-01-15T04:22:53","guid":{"rendered":"http:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/?page_id=74"},"modified":"2020-11-13T03:59:47","modified_gmt":"2020-11-13T03:59:47","slug":"test-plan","status":"publish","type":"page","link":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/test-plan\/","title":{"rendered":"Test Plan"},"content":{"rendered":"

Click here for our Fall Test Plan document<\/a><\/h4>\n

FVD EXPERIMENTS<\/strong><\/h2>\n

1. <\/strong>Validate end-to-end AutoValet system in Gazebo<\/strong><\/p>\n

    \n
  • \n
      \n
    • Location: Remote\/CMU East Garage<\/span><\/li>\n
    • Equipment: Laptop, Gazebo<\/span><\/li>\n
    • Procedure:<\/span>\n
        \n
      • The robot is teleoperated in simulation from start to end and we time the run to act as a baseline for comparison with autonomous operation<\/li>\n
      • Launch all subsystems<\/li>\n
      • The robot will traverse the parking lot, generating waypoints that lie within the lane and traversing
        to those waypoints<\/li>\n
      • When an Aruco tag is detected, the system will calculate a corresponding parking goal pose based on
        the pose of the tag<\/li>\n
      • The robot will perform the parking maneuver and the system will stop<\/li>\n<\/ul>\n<\/li>\n
      • Validation Criteria: The system runs to completion in less than twice the time for teleoperation of the robot in the same path (M.P.1)
        The final parking error should be less than 75 cms in translation and +\/-10 degrees in orientation (M.P.7)
        Success Rate: 90% [Success if M.P.1 or M.P.7 met in 9 out of 10 runs]<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n

        2. Validate End-to-End AutoValet system on the Husky<\/strong><\/p>\n

          \n
        • \n
            \n
          • Location: CMU East Garage<\/span><\/li>\n
          • Equipment: Husky + Sensors, Laptop, Tape measure, Digital
            protractor, Seat belt, Stopwatch
            <\/span><\/li>\n
          • Procedure:<\/span>\n
              \n
            • Seat belt roll is flattened on lane line over the entire run length of the test<\/li>\n
            • Aruco marker placed upright at a spot after the turn looking towards the lane<\/li>\n
            • Robot starts 5 metres before a turn in the center of the lane<\/li>\n
            • We teleoperate the robot to the parking spot and record the time taken<\/li>\n
            • We reset the scene and place the robot back at the start point<\/li>\n
            • Full system is launched. Robot moves along the lane, detects the tag and tries to park<\/li>\n
            • We mark the final pose of the robot footprint on the ground. Record final pose and orientation<\/li>\n<\/ul>\n<\/li>\n
            • Validation Criteria: The time taken for the autonomous parking should be less than twice the time taken while parking with teleoperation (M.P.1)
              The final parking error should be less than 75 cms in translation and +\/-10 degrees in orientation (M.P.7)
              Success Rate: 50% [Success if M.P.1 or M.P.7 met in 2 out of 4 runs]<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n

              Fall Test Plan<\/h2>\n\n\n\n\n\t\n\n\t\n\t\n\t\n\t\n\t\n\t
              MILESTONE<\/th>DELIVERABLE<\/th>TEST METHOD<\/th>\n<\/tr>\n<\/thead>\n
              Mid-September<\/td>Final hardware integration of SLAM and lane detection<\/td>Improve localization and train\/validate detection network on parking lot data<\/td>\n<\/tr>\n
              Late-September<\/td>Navigation on hardware<\/td>Execute point-to-point navigation on the Husky<\/td>\n<\/tr>\n
              Mid-October<\/td>Exploration in simulation<\/td>Introduce the segmented lane information into the 2D costmap and generate waypoints that lie in this lane<\/td>\n<\/tr>\n
              Late-October<\/td>Exploration on Husky<\/td>Show exploration algorithm working on parking lot data<\/td>\n<\/tr>\n
              Mid-November<\/td>Parking maneuver<\/td>Robot will park itself in the target parking spot within 50 cm and 3 degrees of the target pose<\/td>\n<\/tr>\n
              Late-November<\/td>Integration and testing<\/td>Full system validation on hardware<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n

              SVD EXPERIMENTS<\/strong><\/h2>\n\n\n\n

              1. Localization Error Testing of SLAM Subsystem<\/strong><\/p>\n\n\n\n

              • Location: Gazebo Simulation (Live Demo)<\/li>
              • Equipment: Laptop<\/li>
              • Procedure:
                • \u25a0 The robot starts 1m before the start of the turn in our simulated parking garage and the validation
                  script is initiated.
                  \u25a0 User tele-operates the robot around the turn to a point 1m after the end of the turn.
                  \u25a0 The script samples the error in x,y and yaw for every 0.3m moved by the robot in gazebo.
                  \u25a0 A plot of instantaneous error over sample points and average error at end of run is reported.<\/li><\/ul><\/li>
                • Validation criteria: Mean translational error should be less than 50 cm and mean rotational error
                  should be less than 5 degrees at discrete test positions (M.P.4)<\/li><\/ul>\n\n\n\n

                  2. Mapping Error Test<\/strong><\/p>\n\n\n\n

                  • Location: Gazebo Simulation<\/li>
                  • Equipment: Laptop<\/li>
                  • Procedure:
                    • \u25a0 Place 4 boxes (1m x 1m) in the parking lot world in Gazebo
                      \u25a0 Tele-operate the robot in a loop around the simulated environment to generate a map
                      \u25a0 Inspect generated map and record estimated pose of box corners in the grid map manually (RViz)
                      \u25a0 Determine map error by averaging over all discrete pose errors of boxes<\/li><\/ul><\/li>
                    • Validation criteria: The mean error across all points should be within 20 cm (M.P.2)<\/li><\/ul>\n\n\n\n

                      3. Lane Detection Test<\/strong><\/p>\n\n\n\n

                      • Location: Gazebo simulation (Video Demo over the validation set of images)<\/li>
                      • Equipment: Laptop<\/li>
                      • Procedure:
                        • \u25a0 The Gazebo world is modified so that the ground-truth lane is colored green. The robot is tele-operated over this \u201cmodified world\u201d and the camera feed is recorded. The ground truth lane segment is extracted via HSV separation for validation.
                          \u25a0 For testing, we extract the original natural image feed of the camera from the \u201cmodified images\u201d and pass it through our network to get a prediction of the lane.
                          \u25a0 We compare the generated ground truth mask with the network output to calculate the Intersection over Union (IOU) score for the segmentation result and also the mean IOU over all frames.<\/li><\/ul><\/li>
                        • Validation criteria: Mean Intersection over Union (mIOU) score of ego-lane over all frames in the validation set should be at least 60% (M.P.5) (reference)<\/li><\/ul>\n\n\n

                          Spring Milestones<\/h2>\n\n\n\n\n\t\n\n\t\n\t\n\t\n\t\n\t\n\t
                          DATE<\/th>MILESTONE<\/th>DATE<\/th>MILESTONE<\/th>\n<\/tr>\n<\/thead>\n
                          Jan 30<\/td>Test Environment Set up<\/td>Mar 31<\/td>SLAM & Detection subsystem tested on simulation<\/td>\n<\/tr>\n
                          Feb 19<\/td>Progress Review 1<\/td>Apr 8<\/td>Progress Review 4<\/td>\n<\/tr>\n
                          Feb 29<\/td>SLAM on simulation with teleoperation<\/td>Apr 22<\/td>Spring Validation Demo (SVD)<\/td>\n<\/tr>\n
                          Mar 4<\/td>Progress Review 2<\/td>Apr 29<\/td>SVD Encore<\/td>\n<\/tr>\n
                          Mar 20<\/td>Preliminary Design Review<\/td>May 4<\/td>Critical Design Review (CDR)<\/td>\n<\/tr>\n
                          Mar 25<\/td>Progress Review 3<\/td>May 7<\/td>CDR Report<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n","protected":false},"excerpt":{"rendered":"

                          Click here for our Fall Test Plan document FVD EXPERIMENTS 1. Validate end-to-end AutoValet system in Gazebo Location: Remote\/CMU East Garage Equipment: […]<\/p>\n","protected":false},"author":218,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"page-templates\/page_fullwidth.php","meta":{"footnotes":""},"class_list":["post-74","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/wp-json\/wp\/v2\/pages\/74","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/wp-json\/wp\/v2\/users\/218"}],"replies":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/wp-json\/wp\/v2\/comments?post=74"}],"version-history":[{"count":16,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/wp-json\/wp\/v2\/pages\/74\/revisions"}],"predecessor-version":[{"id":665,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/wp-json\/wp\/v2\/pages\/74\/revisions\/665"}],"wp:attachment":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/wp-json\/wp\/v2\/media?parent=74"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}