{"id":384,"date":"2018-02-23T04:52:08","date_gmt":"2018-02-23T04:52:08","guid":{"rendered":"http:\/\/mrsdprojects.ri.cmu.edu\/2017teame\/?page_id=384"},"modified":"2018-04-03T19:52:45","modified_gmt":"2018-04-03T19:52:45","slug":"spring-test-plan","status":"publish","type":"page","link":"https:\/\/mrsdprojects.ri.cmu.edu\/2017teame\/system-implementation\/spring-test-plan\/","title":{"rendered":"Spring Test Plan"},"content":{"rendered":"

Tests<\/span><\/h1>\n

Subsystem 1 – Vision Pipeline<\/span><\/h1>\n

1.1 – Fall system embedding<\/span><\/h2>\n

Objectives:<\/span><\/p>\n

    \n
  • To demonstrate the perception pipeline (detect, track, predict multiple pedestrian) can work on the Jetson TX2.<\/span><\/li>\n<\/ul>\n

    Elements:<\/span><\/p>\n

      \n
    • Jetson TX2, Lidar<\/span><\/li>\n
    • Sensor Drivers, System Libraries and Configurations<\/span><\/li>\n<\/ul>\n

      Location:<\/span><\/p>\n

        \n
      • Basement of Newell-Simon Hall<\/span><\/li>\n<\/ul>\n

        Equipment:<\/span><\/p>\n

          \n
        • Infrastructure (LiDAR, Jetson TX2)<\/span><\/li>\n<\/ul>\n

          Procedure: <\/span><\/p>\n

            \n
          1. Detection<\/span>\n
              \n
            • Place sensor infrastructure 8 meters south and 4 meters east of SE corner of cage. \u00a0Set height to minimum, angle at 0 degrees to horizontal, laser pointing west.<\/span><\/li>\n
            • Measure 6 points from origin. \u00a0Mark these points on the floor with blue tape.<\/span><\/li>\n
            • Measure error against ground truth for first 5 points.<\/span><\/li>\n<\/ul>\n<\/li>\n
            • Tracking<\/span>\n
                \n
              • Human agent walks between (0,20) and (-4,20) repeatedly 5 times.<\/span><\/li>\n<\/ul>\n<\/li>\n
              • Prediction<\/span>\n
                  \n
                • Human agent walks continuously around workspace with prediction system running for one minute.. Compare detected point against predicted trajectory 1.2 seconds before.<\/span><\/li>\n<\/ul>\n<\/li>\n<\/ol>\n

                  Verification<\/span><\/p>\n

                    \n
                  1. Detection<\/span>\n
                      \n
                    • Measure error from ground truth at each point. \u00a0Test passes if average error is less than 0.3 meters.<\/span><\/li>\n<\/ul>\n<\/li>\n
                    • Tracking<\/span>\n
                        \n
                      • Display recorded pedestrian trajectory on desktop.<\/span><\/li>\n
                      • Test fails if no centroid is apparent on the desktop.<\/span><\/li>\n
                      • Test fails if centroid extrema are less than 3.4m apart for 3 or fewer of the cycles.<\/span><\/li>\n<\/ul>\n<\/li>\n
                      • Prediction<\/span>\n
                          \n
                        • View timestamped path to test that trajectory goes full 1.2 seconds into future.<\/span><\/li>\n
                        • Test passes if median error is less than 0.5m for all timesteps over 1 minute.<\/span><\/li>\n<\/ul>\n<\/li>\n<\/ol>\n

                           <\/p>\n

                          1.2 – Multiple pedestrian basics<\/span><\/h2>\n

                          Objectives:<\/span><\/p>\n

                            \n
                          • To demonstrate the single infrastructure based on Jetson TX2 and LiDAR can detect, track, and predict multiple pedestrian with at least 1 meter separated.<\/span><\/li>\n<\/ul>\n

                            Elements:<\/span><\/p>\n

                              \n
                            • Point cloud clustering detection algorithm<\/span><\/li>\n
                            • Hungarian and kalman filter tracking algorithm<\/span><\/li>\n
                            • Social LSTM\/polynomial regression prediction algorithm<\/span><\/li>\n<\/ul>\n

                              Location:<\/span><\/p>\n

                                \n
                              • East garage parking lot.<\/span><\/li>\n<\/ul>\n

                                Equipment:<\/span><\/p>\n

                                  \n
                                • Infrastructure (LiDAR, Jetson TX2, Tripod)<\/span><\/li>\n
                                • Laptop<\/span><\/li>\n<\/ul>\n

                                  Personnel:<\/span><\/p>\n

                                    \n
                                  • All members of Team E<\/span><\/li>\n<\/ul>\n

                                    Procedure:<\/span><\/p>\n

                                      \n
                                    1. Detection<\/span>\n
                                        \n
                                      • Set up infrastructure in open area of garage and define point as origin.<\/span><\/li>\n
                                      • Measure certain points from origin. Any two points are at least 1 meter apart. Mark these points on the floor with blue tape.<\/span><\/li>\n
                                      • 4 human agents stand at 4 different marks. Display centroids on laptop. Measure error as distance of detected centroid coordinates and marked coordinates.<\/span><\/li>\n
                                      • Repeat step 3 for other set of points. Error will be measured from all the points.<\/span><\/li>\n<\/ul>\n<\/li>\n
                                      • Tracking<\/span>\n
                                          \n
                                        • Each human agent walks continuously in test area. Visualize and distinguish each pedestrian by different color in RVIZ. \u00a0All pedestrians maintain 1 meter spacing.<\/span><\/li>\n<\/ul>\n<\/li>\n
                                        • Prediction<\/span>\n
                                            \n
                                          • 4 human agents walk continuously in area. Record fifty trajectory predictions and measured against detection 1.2 seconds in the future. \u00a0Maintain spacing as above.<\/span><\/li>\n<\/ul>\n<\/li>\n<\/ol>\n

                                            Verification Criteria:<\/span><\/p>\n

                                              \n
                                            1. Detection: measure error from ground truth at each point.<\/span>\n
                                                \n
                                              • Test passes if average measure error is equal to or less than 0.3 meters.<\/span><\/li>\n
                                              • Test fails if average measure error is more than 0.3 meters.<\/span><\/li>\n<\/ul>\n<\/li>\n
                                              • Tracking: display recorded pedestrians\u2019 trajectories on desktop.<\/span>\n
                                                  \n
                                                • Test passes if all pedestrians\u2019 ids are consistent all the time.<\/span><\/li>\n
                                                • Test fails if there are inconsistent tracking id for any pedestrian.<\/span><\/li>\n<\/ul>\n<\/li>\n
                                                • Prediction: measure predicted position 1.2s in the future against detected position.<\/span>\n
                                                    \n
                                                  • Test passes if average error for each pedestrian\u2019s test is less than 0.5 meters.<\/span><\/li>\n
                                                  • Test fails if average error for one pedestrian\u2019s test is more than 0.5 meters.<\/span><\/li>\n<\/ul>\n<\/li>\n<\/ol>\n

                                                    1.3 – Multiple pedestrian advanced<\/span><\/h2>\n

                                                    Objectives:<\/span><\/p>\n

                                                      \n
                                                    • To demonstrate the single infrastructure based on Jetson TX2, LiDAR and camera can detect, track, and predict multiple pedestrian with no spacing constraint.<\/span><\/li>\n<\/ul>\n

                                                      Elements:<\/span><\/p>\n

                                                        \n
                                                      • Point cloud clustering and camera human detection algorithm<\/span><\/li>\n
                                                      • Hungarian and kalman filter tracking algorithm<\/span><\/li>\n
                                                      • Social LSTM\/polynomial regression prediction algorithm<\/span><\/li>\n<\/ul>\n

                                                        Location:<\/span><\/p>\n

                                                          \n
                                                        • East garage parking lot<\/span><\/li>\n<\/ul>\n

                                                          Equipment:<\/span><\/p>\n

                                                            \n
                                                          • Infrastructure (LiDAR, Camera, Jetson TX2, Tripod)<\/span><\/li>\n
                                                          • Laptop<\/span><\/li>\n<\/ul>\n

                                                            Personnel:<\/span><\/p>\n

                                                              \n
                                                            • All<\/span><\/li>\n<\/ul>\n

                                                              Procedure:<\/span><\/p>\n

                                                                \n
                                                              1. Detection<\/span>\n
                                                                  \n
                                                                • Set up infrastructure in open area of garage and define point as origin.<\/span><\/li>\n
                                                                • Measure certain points from origin. Mark these points on the floor with blue tape.<\/span><\/li>\n
                                                                • 4 human agents stand at 4 different marks. Display centroids on laptop. Measure error as distance of detected centroid coordinates and marked coordinates.<\/span><\/li>\n
                                                                • Repeat step 3 for other set of points. Error will be measured from all the points.<\/span><\/li>\n<\/ul>\n<\/li>\n
                                                                • Tracking<\/span>\n
                                                                    \n
                                                                  • Each human agent walks continuously in test area. Visualize and distinguish each pedestrian by different color in RVIZ. <\/span><\/li>\n<\/ul>\n<\/li>\n
                                                                  • Prediction<\/span>\n
                                                                      \n
                                                                    • 4 human agents walk continuously in test area. Capture fifty trajectory predictions and measured against detection 1.2 seconds in the future.<\/span><\/li>\n<\/ul>\n<\/li>\n<\/ol>\n

                                                                      Verification Criteria:<\/span><\/p>\n

                                                                        \n
                                                                      1. Detection: measure error from ground truth at each point.<\/span>\n
                                                                          \n
                                                                        • Test passes if average measure error is equal to or less than 0.3 meters.<\/span><\/li>\n
                                                                        • Test fails if average measure error is more than 0.3 meters.<\/span><\/li>\n<\/ul>\n<\/li>\n
                                                                        • Tracking: display recorded pedestrians\u2019 trajectories on desktop.<\/span>\n
                                                                            \n
                                                                          • Test passes if all pedestrians\u2019 ids are consistent all the time.<\/span><\/li>\n
                                                                          • Test fails if there are inconsistent tracking id for any pedestrian.<\/span><\/li>\n<\/ul>\n<\/li>\n
                                                                          • Prediction: measure predicted position 1.2s in the future against detected position.<\/span>\n
                                                                              \n
                                                                            • Test passes if average error for each pedestrian\u2019s test is less than 0.5 meters.<\/span><\/li>\n
                                                                            • Test fails if average error for one pedestrian\u2019s test is more than 0.5 meters.<\/span><\/li>\n<\/ul>\n<\/li>\n<\/ol>\n

                                                                              1.4 – Coverage of full intersection<\/span><\/h2>\n

                                                                              Objectives:<\/span><\/p>\n

                                                                                \n
                                                                              • To demonstrate the perception system (with two infrastructures and two Jetson TX2) can detect pedestrians at every point within the perimeter of the intersection.<\/span><\/li>\n<\/ul>\n

                                                                                Elements:<\/span><\/p>\n

                                                                                  \n
                                                                                • Detection around perimeter<\/span><\/li>\n<\/ul>\n

                                                                                  Location:<\/span><\/p>\n

                                                                                    \n
                                                                                  • East garage parking lot<\/span><\/li>\n<\/ul>\n

                                                                                    Equipment:<\/span><\/p>\n

                                                                                      \n
                                                                                    • Infrastructure (LiDAR, Jetson TX2, Tripod, Cameras)<\/span><\/li>\n
                                                                                    • Power Supply<\/span><\/li>\n<\/ul>\n

                                                                                      Personnel:<\/span><\/p>\n

                                                                                        \n
                                                                                      • All members of Team E<\/span><\/li>\n<\/ul>\n

                                                                                        Procedure:<\/span><\/p>\n

                                                                                          \n
                                                                                        1. Infrastructure Setup<\/span>\n
                                                                                            \n
                                                                                          • Place sensor infrastructure A \u00a0and B on opposite corners of the intersection. Set height to minimum, angle at 0 degrees to horizontal.<\/span><\/li>\n<\/ul>\n<\/li>\n
                                                                                          • Detection<\/span>\n
                                                                                              \n
                                                                                            • Measure certain points from origin (Infrastructure A) within 20 meters range. Any two points are at least 1 meter apart. Mark these points on the floor with blue tape.<\/span><\/li>\n
                                                                                            • 4 human agents stand at 4 different marks. Display centroid on desktop. Measure error as distance from the origin point.<\/span><\/li>\n
                                                                                            • Repeat step b for other set of points. Error will be measured from all the points.<\/span><\/li>\n<\/ul>\n<\/li>\n<\/ol>\n

                                                                                               <\/p>\n

                                                                                              Verification Criteria:<\/span><\/p>\n

                                                                                                \n
                                                                                              1. Make sure the infrastructures are calibrated and the system is functioning<\/span><\/li>\n
                                                                                              2. Detection: measure error from ground truth at each point.<\/span>\n
                                                                                                  \n
                                                                                                • Test passes if average measure error is equal to or less than 0.3 meters.<\/span><\/li>\n
                                                                                                • Test fails if average measure error is more than 0.3 meters.<\/span><\/li>\n<\/ul>\n<\/li>\n<\/ol>\n

                                                                                                   <\/p>\n

                                                                                                  1.5 – Cycle time within specification<\/span><\/h2>\n

                                                                                                  Objectives:<\/span><\/p>\n

                                                                                                    \n
                                                                                                  • To ensure that the cycle time of our complete system meets the corresponding performance requirement<\/span><\/li>\n<\/ul>\n

                                                                                                    Elements:<\/span><\/p>\n

                                                                                                      \n
                                                                                                    • Entire perception system<\/span><\/li>\n<\/ul>\n

                                                                                                      Location:<\/span><\/p>\n

                                                                                                        \n
                                                                                                      • Gesling Parking<\/span><\/li>\n<\/ul>\n

                                                                                                        Equipment:<\/span><\/p>\n

                                                                                                          \n
                                                                                                        • Two Infrastructures, LiDARs, Jetson TX2<\/span><\/li>\n
                                                                                                        • Cardboard boxes to serve as occlusion<\/span><\/li>\n<\/ul>\n

                                                                                                          Personnel:<\/span><\/p>\n

                                                                                                            \n
                                                                                                          • All members of Team E<\/span><\/li>\n<\/ul>\n

                                                                                                            Procedure:<\/span><\/p>\n

                                                                                                              \n
                                                                                                            1. A pedestrian will stand behind an obstacle (stack of cardboard boxes)<\/span><\/li>\n
                                                                                                            2. System will be restarted<\/span><\/li>\n
                                                                                                            3. Pedestrian will walk out from behind obstacle and enter range of infrastructures<\/span><\/li>\n
                                                                                                            4. Timestamp will be recorded when pedestrian is first detected<\/span><\/li>\n
                                                                                                            5. Timestamp will be recorded when predicted pedestrian trajectory is first published<\/span><\/li>\n<\/ol>\n

                                                                                                              Verification Criteria:<\/span><\/p>\n

                                                                                                                \n
                                                                                                              • Difference between the two recorded timestamps must be less than 0.5s<\/span><\/li>\n<\/ul>\n

                                                                                                                 <\/p>\n

                                                                                                                Subsystem 2 – Communication<\/span><\/h1>\n

                                                                                                                2.1 – Hardware Communication<\/span><\/h2>\n

                                                                                                                Objectives:<\/span><\/p>\n

                                                                                                                  \n
                                                                                                                • To demonstrate the wireless communication between Jetson TX2 and onboard computer.<\/span><\/li>\n<\/ul>\n

                                                                                                                  Elements:<\/span><\/p>\n

                                                                                                                    \n
                                                                                                                  • Zigbee enabled wireless communication<\/span><\/li>\n<\/ul>\n

                                                                                                                    Location:<\/span><\/p>\n

                                                                                                                      \n
                                                                                                                    • Basement of Newell-Simon Hall<\/span><\/li>\n<\/ul>\n

                                                                                                                      Equipment:<\/span><\/p>\n

                                                                                                                        \n
                                                                                                                      • Onboard computer (Arduino with Xbee shield, Xbee) attached to laptop<\/span><\/li>\n
                                                                                                                      • Jetson TX2 attached to monitor and keyboard<\/span><\/li>\n<\/ul>\n

                                                                                                                        Personnel:<\/span><\/p>\n

                                                                                                                          \n
                                                                                                                        • Oliver Krengel<\/span><\/li>\n<\/ul>\n

                                                                                                                          Procedure:<\/span><\/p>\n

                                                                                                                            \n
                                                                                                                          1. Power on Jetson TX2 connected to monitor and keyboard<\/span><\/li>\n
                                                                                                                          2. Power on Arduino via laptop connection<\/span><\/li>\n
                                                                                                                          3. Open communication windows on Arduino and Jetson<\/span><\/li>\n
                                                                                                                          4. Enter string input on laptop e.g. \u201cHello infrastructure\u201d<\/span><\/li>\n
                                                                                                                          5. Observe string from step 4 on monitor<\/span><\/li>\n
                                                                                                                          6. Enter string input on keyboard attached to Jetson TX2<\/span><\/li>\n
                                                                                                                          7. Observe string input from step 6 on laptop<\/span><\/li>\n<\/ol>\n

                                                                                                                            Verification Criteria:<\/span><\/p>\n

                                                                                                                              \n
                                                                                                                            • String input from step 4 is observed on monitor in step 5<\/span><\/li>\n
                                                                                                                            • String input from step 6 is observed on laptop in step 7<\/span><\/li>\n
                                                                                                                            • Two-way communication is verified if criteria 1 and 2 are both met<\/span><\/li>\n<\/ul>\n

                                                                                                                               <\/p>\n

                                                                                                                              2.2 GPS-based Localization<\/span><\/h2>\n

                                                                                                                              Objectives:<\/span><\/p>\n

                                                                                                                                \n
                                                                                                                              • To ensure that the vehicle is aware of its position at a speed of 20-30 mph<\/span><\/li>\n<\/ul>\n

                                                                                                                                Elements:<\/span><\/p>\n

                                                                                                                                  \n
                                                                                                                                • Vehicle localization<\/span><\/li>\n<\/ul>\n

                                                                                                                                  Location:<\/span><\/p>\n

                                                                                                                                    \n
                                                                                                                                  • Schenley Drive (between Phipps and Schenley Plaza)<\/span><\/li>\n<\/ul>\n

                                                                                                                                    Equipment:<\/span><\/p>\n

                                                                                                                                      \n
                                                                                                                                    • Reach RTK GPS Kit with Base Station <\/span><\/li>\n
                                                                                                                                    • The Vehicle<\/span><\/li>\n<\/ul>\n

                                                                                                                                      Personnel:<\/span><\/p>\n

                                                                                                                                        \n
                                                                                                                                      • Oliver Krengel<\/span><\/li>\n
                                                                                                                                      • Rohit Murthy<\/span><\/li>\n<\/ul>\n

                                                                                                                                        Procedure:<\/span><\/p>\n

                                                                                                                                          \n
                                                                                                                                        1. Identify the exact GPS points of Schenley Plaza and Phipps on Schenley Drive by averaging the results from our GPS module over half an hour to serve as \u2018ground truth<\/span><\/li>\n
                                                                                                                                        2. Drive from start to finish by reaching speeds between 20mph and 30mph<\/span><\/li>\n
                                                                                                                                        3. Record the GPS coordinates returned by GPS module<\/span><\/li>\n
                                                                                                                                        4. Compare with \u2018ground truth\u2019 identified earlier<\/span><\/li>\n
                                                                                                                                        5. Repeat 3 times<\/span><\/li>\n<\/ol>\n

                                                                                                                                          Verification Criteria:<\/span><\/p>\n

                                                                                                                                            \n
                                                                                                                                          • Difference between real-time GPS points and \u2018ground truth\u2019 < 0.5m for all 3 attempts<\/span><\/li>\n<\/ul>\n

                                                                                                                                             <\/p>\n

                                                                                                                                            2.3 – GUI for driver<\/span><\/h2>\n

                                                                                                                                            Objective:<\/span><\/p>\n

                                                                                                                                              \n
                                                                                                                                            • To verify that GUI is able to provide relevant information to the car\/driver to avoid collision with the pedestrians.<\/span><\/li>\n<\/ul>\n

                                                                                                                                              Elements:<\/span><\/p>\n

                                                                                                                                                \n
                                                                                                                                              • GUI<\/span><\/li>\n
                                                                                                                                              • Infrastructure<\/span><\/li>\n<\/ul>\n

                                                                                                                                                Procedure:<\/span><\/p>\n

                                                                                                                                                  \n
                                                                                                                                                1. Power on the infrastructure.<\/span><\/li>\n
                                                                                                                                                2. Visualize the output of all the sensors in Rviz.<\/span><\/li>\n
                                                                                                                                                3. Make sure the connection is established with the infrastructure<\/span><\/li>\n
                                                                                                                                                4. A message would be sent to the GUI from the infrastrucuture like for example- \u201cTest GUI\u201d.<\/span><\/li>\n
                                                                                                                                                5. Open the GUI and wait for the message to be received.<\/span><\/li>\n<\/ol>\n

                                                                                                                                                  Verification:<\/span><\/p>\n

                                                                                                                                                    \n
                                                                                                                                                  1. The \u00a0test for GUI would be successful if the GUI receives the message sent from the infrastructure.<\/span><\/li>\n<\/ol>\n

                                                                                                                                                     <\/p>\n

                                                                                                                                                    Subsystem 3 – Infrastructure<\/span><\/h1>\n

                                                                                                                                                    3.1 – Infrastructure completion<\/span><\/h2>\n

                                                                                                                                                    Objective:<\/span><\/p>\n

                                                                                                                                                      \n
                                                                                                                                                    • To verify that the infrastructure is complete and is able to perform all the related functions satisfactorily.<\/span><\/li>\n<\/ul>\n

                                                                                                                                                      Elements:<\/span><\/p>\n

                                                                                                                                                        \n
                                                                                                                                                      • Power Distribution Board.<\/span><\/li>\n
                                                                                                                                                      • Sensor Mount.<\/span><\/li>\n
                                                                                                                                                      • Sensors -LIDAR and Camera.<\/span><\/li>\n<\/ul>\n

                                                                                                                                                        Personnel:<\/span><\/p>\n

                                                                                                                                                          \n
                                                                                                                                                        • Vivek Gr<\/span><\/li>\n<\/ul>\n

                                                                                                                                                          Procedures:<\/span><\/p>\n

                                                                                                                                                            \n
                                                                                                                                                          1. Power the PDB and test all the input voltage for the sensors by using multimeter.<\/span><\/li>\n
                                                                                                                                                          2. Unplug the PDB from the battery.<\/span><\/li>\n
                                                                                                                                                          3. Complete all connections between the PDB and the sensors (LIDAR and the camera).<\/span><\/li>\n
                                                                                                                                                          4. Power the board again and notice the LEDs glowing up.<\/span><\/li>\n
                                                                                                                                                          5. Connect the interfacing cable of both the LIDAR and the camera to the Jetson Tx2.<\/span><\/li>\n<\/ol>\n

                                                                                                                                                            Verification:<\/span><\/p>\n