{"id":153,"date":"2019-02-22T02:34:43","date_gmt":"2019-02-22T02:34:43","guid":{"rendered":"http:\/\/mrsdprojects.ri.cmu.edu\/2018teame\/?page_id=153"},"modified":"2019-10-02T00:03:40","modified_gmt":"2019-10-02T00:03:40","slug":"test-plan","status":"publish","type":"page","link":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teame\/project-management\/test-plan\/","title":{"rendered":"Test Plan"},"content":{"rendered":"

Fall Validation Experiments<\/b><\/h1>\n

We have eight validation tests planned in Fall 2019, these include sub-system level tests for monitoring and navigation sub-systems and complete system-level tests as well.<\/p>\n

 <\/p>\n\n\n\n\n\n\n\n\n\n
Progress Review<\/b><\/td>\nCapability Milestones<\/b><\/td>\nAssociated Tests<\/b><\/td>\nAssociated Requirements<\/b><\/td>\n<\/tr>\n
#8 (9\/25)<\/span><\/td>\n\n
    \n
  • Visualizer V1 complete<\/span><\/li>\n
  • RTK Node complete<\/span><\/li>\n<\/ul>\n<\/td>\n
<\/td>\n<\/td>\n<\/tr>\n
#9 (10\/9)<\/span><\/td>\n\n
    \n
  • MVP of Monitoring complete<\/span><\/li>\n
  • Localization integrated<\/span><\/li>\n
  • Camera exposure v1 complete<\/span><\/li>\n<\/ul>\n<\/td>\n
Test 05<\/span><\/td>\nMN1,2<\/span><\/td>\n<\/tr>\n
#10 (10\/23)<\/span><\/td>\n\n
    \n
  • Navigation MVP complete<\/span><\/li>\n<\/ul>\n<\/td>\n
Test 03, 04, 07<\/span><\/td>\nMR 3,4,5,6 (MR 6.1 – 6.4)<\/span><\/td>\n<\/tr>\n
#11 (11\/6)<\/span><\/td>\n\n
    \n
  • Wheel covers installed<\/span><\/li>\n<\/ul>\n<\/td>\n
Test 01, 02, 06<\/span><\/td>\nMR 1, MR 2<\/span><\/p>\n

MN 4<\/span><\/td>\n<\/tr>\n

#12 (11\/18)<\/span><\/td>\n\n
    \n
  • Integration and Testing<\/span><\/li>\n<\/ul>\n<\/td>\n
<\/td>\n<\/td>\n<\/tr>\n
FVD (11\/25)<\/span><\/td>\n\n
    \n
  • Full system integration<\/span><\/li>\n<\/ul>\n<\/td>\n
Test 08<\/span><\/td>\nMR 1-6<\/span><\/p>\n

MN 1-4<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Note:<\/b> Generally, tests are scheduled for the PR after the capability milestone has been achieved to leave room for finishing touches and dry runs before presenting the results.\u00a0<\/span><\/p>\n

 <\/p>\n

Tests<\/span><\/h1>\n

 <\/p>\n\n\n\n\n\n\n\n\n\n
Test Number:<\/b><\/td>\n01<\/span><\/td>\nTest Name:<\/b><\/td>\n\n

Autonomous Navigation Test<\/span><\/h2>\n<\/td>\n

Test Date:<\/b><\/td>\n11\/6<\/span><\/td>\n<\/tr>\n
Objective:<\/b><\/td>\nAutonomously traverse field without hitting plant stem (MN4, MR1, MR2)<\/span><\/td>\n<\/tr>\n
Elements Tested:<\/b><\/td>\nSubsystem: Autonomous Row Navigation<\/span><\/td>\n<\/tr>\n
Location:<\/b><\/td>\nRivendale Farms – Recorded on Video<\/span><\/td>\n<\/tr>\n
Equipment:<\/b><\/td>\n\n
    \n
  • See field visit checklist<\/span><\/li>\n
  • Map of the field<\/span><\/li>\n
  • Cameras<\/span><\/li>\n<\/ul>\n<\/td>\n
Personnel:<\/b><\/td>\n\n
    \n
  • Team E<\/span><\/li>\n
  • Rivendale farm representatives (Not required)<\/span><\/li>\n<\/ul>\n<\/td>\n<\/tr>\n
Procedure:<\/b><\/td>\n\n
    \n
  1. Place the robot at the start of the first row<\/span><\/li>\n
  2. Command the robot to start autonomous navigation<\/span><\/li>\n
  3. Robot traverses the row and switches to the next row<\/span><\/li>\n
  4. The robot stops after traversing 5 rows<\/span><\/li>\n<\/ol>\n<\/td>\n<\/tr>\n
Verification Criteria:<\/b><\/td>\n\n
    \n
  • Robot does not hit plant stems while navigating<\/span><\/li>\n
  • Robot successfully switches row 4 out of 5 times<\/span><\/li>\n<\/ul>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

     <\/p>\n\n\n\n\n\n\n\n\n\n
    Test Number:<\/b><\/td>\n02<\/span><\/td>\nTest Name:<\/b><\/td>\n\n

    Coverage Planner Test<\/span><\/h2>\n<\/td>\n

    		Test Date:<\/b><\/td>\n11\/6<\/span><\/td>\n<\/tr>\n
    Objective:<\/b><\/td>\nVerify coverage plan covers the entire field of interest\u00a0<\/span><\/td>\n<\/tr>\n
    Elements Tested:<\/b><\/td>\nSubsystem: Coverage Planner<\/span><\/td>\n<\/tr>\n
    Location:<\/b><\/td>\nNewell Simon Hall<\/span><\/td>\n<\/tr>\n
    Equipment:<\/b><\/td>\n\n
      \n
    • Map File<\/span><\/li>\n
    • Computer<\/span><\/li>\n<\/ul>\n<\/td>\n
    		Personnel:<\/b><\/td>\n\n
      \n
    • Team E<\/span><\/li>\n
    • MRSD Advisors<\/span><\/li>\n<\/ul>\n<\/td>\n<\/tr>\n
    Procedure:<\/b><\/td>\n\n
      \n
    1. Load the map file<\/span><\/li>\n
    2. Software generates the coverage plan<\/span><\/li>\n
    3. Play animation showing the entire coverage plan<\/span><\/li>\n<\/ol>\n<\/td>\n<\/tr>\n
    Verification Criteria:<\/b><\/td>\n\n
      \n
    • Coverage planner covers the rows of interest that the robot can reach<\/span><\/li>\n<\/ul>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

       <\/p>\n\n\n\n\n\n\n\n\n\n
      Test Number:<\/b><\/td>\n03<\/span><\/td>\nTest Name:<\/b><\/td>\n\n

      Pest\/Disease Perception Software Test<\/span><\/h2>\n<\/td>\n

      		Test Date:<\/b><\/td>\n10\/23<\/span><\/td>\n<\/tr>\n
      Objective:<\/b><\/td>\nEvaluate the performance of plant health monitoring deep net (MR4, MR5)<\/span><\/td>\n<\/tr>\n
      Elements Tested:<\/b><\/td>\nSubsystem: Mask-RCNN (Perception)<\/span><\/td>\n<\/tr>\n
      Location:<\/b><\/td>\nNewell Simon Hall<\/span><\/td>\n<\/tr>\n
      Equipment:<\/b><\/td>\n\n
        \n
      • Server with GPU<\/span><\/li>\n
      • Test images (labeled {positive, negative}) from a new spatial area of the field, on which we have not trained before<\/span><\/li>\n<\/ul>\n<\/td>\n
      		Personnel:<\/b><\/td>\n\n
        \n
      • Team E<\/span><\/li>\n
      • MRSD Advisors<\/span><\/li>\n<\/ul>\n<\/td>\n<\/tr>\n
      Procedure:<\/b><\/td>\n\n
        \n
      1. Power on gpu server<\/span><\/li>\n
      2. Run plant health model inference<\/span><\/li>\n
      3. Software loads test images and computes performance metric<\/span><\/li>\n<\/ol>\n<\/td>\n<\/tr>\n
      Verification Criteria:<\/b><\/td>\n\n
        \n
      • Robot successfully identifies fungus and holes<\/span><\/li>\n<\/ul>\n

        with greater than 80% precision and recall with unhealthy defined as positive\u00a0<\/span><\/p>\n

        \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0* precision = Tp \/ (Fp + Tp), recall = Tp \/ (Fn + Tp)<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

         <\/p>\n\n\n\n\n\n\n\n\n\n
        Test Number:<\/b><\/td>\n04<\/span><\/td>\nTest Name:<\/b><\/td>\nVisualization Subsystem Test<\/span><\/td>\nTest Date:<\/b><\/td>\n10\/23<\/span><\/td>\n<\/tr>\n
        Objective:<\/b><\/td>\nEvaluate the speed of monitoring pipeline and GUI Features (MR6)<\/span><\/td>\n<\/tr>\n
        Elements Tested:<\/b><\/td>\nSubsystem: Monitoring pipeline\u00a0<\/span><\/td>\n<\/tr>\n
        Location:<\/b><\/td>\nNewell Simon Hall<\/span><\/td>\n<\/tr>\n
        Equipment:<\/b><\/td>\n\n
          \n
        • GPU Server\u00a0<\/span><\/li>\n
        • Pre-recorded ROS bag files of the entire rows of interest<\/span><\/li>\n<\/ul>\n<\/td>\n
        		Personnel:<\/b><\/td>\n\n
          \n
        • Team E<\/span><\/li>\n
        • MRSD Advisors<\/span><\/li>\n<\/ul>\n<\/td>\n<\/tr>\n
        Procedure:<\/b><\/td>\n\n
          \n
        1. Power on GPU server<\/span><\/li>\n
        2. Run visualization pipeline<\/span><\/li>\n
        3. Software loads test images and visualizes the results in GUI<\/span><\/li>\n
        4. Visually demonstrate the field\u2019s layout<\/span><\/li>\n
        5. Select a datapoint and change the classification, to demonstrate change in the stored data<\/span><\/li>\n
        6. End the visualizer and relaunch it to show the same data<\/span><\/li>\n<\/ol>\n<\/td>\n<\/tr>\n
        Verification Criteria:<\/b><\/td>\n\n
          \n
        • Robot successfully processes data at a rate faster<\/span><\/li>\n<\/ul>\n

          than one field per 24 hours (MR 6)<\/span><\/p>\n

            \n
          • Clear depiction of the field layout and data presentation (subjective)<\/span><\/li>\n
          • Successful use of interactive portions of the GUI<\/span><\/li>\n
          • Successful data preservation on relaunch<\/span><\/li>\n<\/ul>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

            Note: <\/b>The GUI features are subject to change, pending product reviews with the farmers<\/span><\/p>\n

             <\/p>\n\n\n\n\n\n\n\n\n\n
            Test Number:<\/b><\/td>\n05<\/span><\/td>\nTest Name:<\/b><\/td>\n\n

            Robot Platform Verification<\/span><\/h2>\n<\/td>\n

            		Test Date:<\/b><\/td>\n10\/9<\/span><\/td>\n<\/tr>\n
            Objective:<\/b><\/td>\nVerify Non-Functional Requirements related to the Robot Platform\u00a0<\/span><\/p>\n

            (MN1, MN2)<\/span><\/td>\n<\/tr>\n

            Elements Tested:<\/b><\/td>\nSubsystem: Robot Platform<\/span><\/td>\n<\/tr>\n
            Location:<\/b><\/td>\nRivendale Farms<\/span><\/td>\n<\/tr>\n
            Equipment:<\/b><\/td>\n\n
              \n
            • Video Camera<\/span><\/li>\n
            • Robot<\/span><\/li>\n
            • Tape Measure<\/span><\/li>\n<\/ul>\n<\/td>\n
            		Personnel:<\/b><\/td>\n\n
              \n
            • Team E<\/span><\/li>\n
            • MRSD Advisors<\/span><\/li>\n<\/ul>\n<\/td>\n<\/tr>\n
            Procedure:<\/b><\/td>\n\n
              \n
            1. Power on robot<\/span><\/li>\n
            2. Drive the robot via joystick<\/span><\/li>\n
            3. Continue to press on joystick, releasing the remote kill switch, demonstrating a stop<\/span><\/li>\n
            4. Drive the robot via joystick<\/span><\/li>\n
            5. Press the mechanical E-Stop to demonstrate a stop<\/span><\/li>\n
            6. Drive the robot to the beginning of the first row of the brassica field<\/span><\/li>\n
            7. Measure the open space between the robot and the plant stems, to demonstrate plant clearance<\/span><\/li>\n<\/ol>\n<\/td>\n<\/tr>\n
            Verification Criteria:<\/b><\/td>\n\n
              \n
            • Effectiveness of remote kill switch<\/span><\/li>\n
            • Effectiveness of mechanical E-stop<\/span><\/li>\n
            • Fit of robot into field\u2019s rows<\/span><\/li>\n<\/ul>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

               <\/p>\n\n\n\n\n\n\n\n\n\n
              Test Number:<\/b><\/td>\n06<\/span><\/td>\nTest Name:<\/b><\/td>\n\n

              Battery Life Test<\/span><\/h2>\n<\/td>\n

              		Test Date:<\/b><\/td>\n11\/6<\/span><\/td>\n<\/tr>\n
              Objective:<\/b><\/td>\nConfirm battery life is sufficient for the Rivendale Brassica Field (MN3)<\/span><\/td>\n<\/tr>\n
              Elements Tested:<\/b><\/td>\nSubsystem: Robot Platform<\/span><\/td>\n<\/tr>\n
              Location:<\/b><\/td>\nRivendale Farms<\/span><\/td>\n<\/tr>\n
              Equipment:<\/b><\/td>\n\n
                \n
              • Robot<\/span><\/li>\n
              • Video camera<\/span><\/li>\n<\/ul>\n<\/td>\n
              		Personnel:<\/b><\/td>\n\n
                \n
              • Team E<\/span><\/li>\n
              • MRSD Advisors<\/span><\/li>\n<\/ul>\n<\/td>\n<\/tr>\n
              Procedure:<\/b><\/td>\n\n
                \n
              1. Place the fully charged robot at the start of the first row<\/span><\/li>\n
              2. Drive the robot at standard speed, through the entire brassica field<\/span><\/li>\n<\/ol>\n<\/td>\n<\/tr>\n
              Verification Criteria:<\/b><\/td>\n\n
                \n
              • Sufficient Battery Life to cover the entire brassica field<\/span><\/li>\n<\/ul>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

                 <\/p>\n\n\n\n\n\n\n\n\n\n
                Test Number:<\/b><\/td>\n07<\/span><\/td>\nTest Name:<\/b><\/td>\n\n

                Usable row images<\/span><\/h2>\n<\/td>\n

                		Test Date:<\/b><\/td>\n10\/27<\/span><\/td>\n<\/tr>\n
                Objective:<\/b><\/td>\nConfirm that we are collecting images of acceptable exposure which are usable for the deep learning pipeline. (MR3)<\/span><\/td>\n<\/tr>\n
                Elements Tested:<\/b><\/td>\nSubsystem: Mask-RCNN (Perception)<\/span><\/td>\n<\/tr>\n
                Location:<\/b><\/td>\nNewell Simon Hall<\/span><\/td>\n<\/tr>\n
                Equipment:<\/b><\/td>\n\n
                  \n
                • ROS Bag<\/span><\/li>\n
                • ROS-enabled laptop<\/span><\/li>\n<\/ul>\n<\/td>\n
                		Personnel:<\/b><\/td>\n\n
                  \n
                • Team E<\/span><\/li>\n
                • MRSD Advisors<\/span><\/li>\n<\/ul>\n<\/td>\n<\/tr>\n
                Procedure:<\/b><\/td>\n\n
                  \n
                1. Take a ROS Bag consisting of left and right camera images collected from a row traversal as input.\u00a0<\/span><\/li>\n
                2. Use the exposure testing script to process images and find the percentage of images that pass an over\/underexposed test.\u00a0<\/span><\/li>\n<\/ol>\n<\/td>\n<\/tr>\n
                Verification Criteria:<\/b><\/td>\n\n
                  \n
                • The percentage of images that pass the test should be > 75%\u00a0<\/span><\/li>\n<\/ul>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

                   <\/p>\n\n\n\n\n\n\n\n\n\n
                  Test Number:<\/b><\/td>\n08<\/span><\/td>\nTest Name:<\/b><\/td>\n\n

                  System Integration Test<\/span><\/h2>\n<\/td>\n

                  		Test Date:<\/b><\/td>\n11\/25<\/span><\/td>\n<\/tr>\n
                  Objective:<\/b><\/td>\nVerify end-to-end robot system functionality ( MR 1-6, MN1-4)<\/span><\/td>\n<\/tr>\n
                  Elements Tested:<\/b><\/td>\nRobot Platform, Navigation, Plant Health Monitoring, GUI<\/span><\/td>\n<\/tr>\n
                  Location:<\/b><\/td>\nRivendale Farms \/ Newell-Simon Hall<\/span><\/td>\n<\/tr>\n
                  Equipment:<\/b><\/td>\nRivendale Farms<\/span><\/p>\n
                    \n
                  • Video Camera<\/span><\/li>\n
                  • Robot<\/span><\/li>\n
                  • Joystick<\/span><\/li>\n
                  • Portable hard drive<\/span><\/li>\n<\/ul>\n

                    Newell-Simon Hall<\/span><\/p>\n

                      \n
                    • Server with GPU<\/span><\/li>\n<\/ul>\n<\/td>\n
                  		Personnel:<\/b><\/td>\n\n
                    \n
                  • Team E (Rivendale Farms)<\/span><\/li>\n
                  • MRSD Advisors (Newell-Simon Hall)<\/span><\/li>\n<\/ul>\n<\/td>\n<\/tr>\n
                  Procedure:<\/b><\/td>\n\n
                    \n
                  1. Set up video camera and begin recording<\/span><\/li>\n
                  2. Complete Test 01 Autonomous Navigation Test<\/span><\/li>\n
                  3. End recording of video<\/span><\/li>\n
                  4. Copy the ROS bag from the plant health monitoring run to the portable hard drive<\/span><\/li>\n
                  5. Transport the hard drive to the MRSD lab and copy files to server for inference<\/span><\/li>\n
                  6. Run Test 07 Usable Row Images Test on server<\/span><\/li>\n
                  7. Run Test 03 Pest\/Disease Perception Software Test<\/span><\/li>\n
                  8. Complete Test 04 GUI Feature Verification for this new data<\/span><\/li>\n<\/ol>\n<\/td>\n<\/tr>\n
                  Verification Criteria:<\/b><\/td>\n\n
                    \n
                  • Video correctly shows criteria for Test 01, Autonomous Navigation passed<\/span><\/li>\n
                  • Collected data passes Test 07 Usable Row Images test<\/span><\/li>\n
                  • System passes Test 03 Pest\/Disease Perception Software Test on newly collected data<\/span><\/li>\n
                  • System passes Test 04 GUI Feature Verification on newly collected data<\/span><\/li>\n<\/ul>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

                    Spring Validation Experiments<\/b><\/h1>\n

                    We have planned five validation tests in the spring of 2019 that are focused primarily on navigation. As such, their success criteria correspond to the navigation-related performance requirements. We have also tried to minimize the number of in-field tests required, to save on time and resources, so tests 3 through 5, which are primarily software systems, will be tested on pre-recorded data.<\/span><\/p>\n

                    Test 1: In-Row Navigation<\/b><\/h2>\n

                    Location:<\/b> Rivendale Farms<\/span><\/p>\n

                    Equipment:<\/b> Robot, 2 rows of plants<\/span><\/p>\n

                    Setup:<\/b><\/p>\n

                      \n
                    • Place robot at the entrance to a row of plants, facing into the row<\/span><\/li>\n
                    • Robot has pre-generated map file<\/span><\/li>\n<\/ul>\n

                      Test:<\/b><\/p>\n

                        \n
                      1. Power on the robot<\/span><\/li>\n
                      2. Establish connection to the robot<\/span><\/li>\n
                      3. Command the robot to traverse the row<\/span><\/li>\n
                      4. Robot navigates along the row<\/span><\/li>\n
                      5. Robot stops at end of row<\/span><\/li>\n<\/ol>\n

                        Success Criteria:<\/b><\/p>\n

                          \n
                        • Robot fits in row (MN1)<\/span><\/li>\n
                        • Robot arrives at the far end of row <\/span><\/li>\n
                        • Robot does not crush or trample any plants (MN5)<\/span><\/li>\n<\/ul>\n

                           <\/p>\n

                          Test 2: Switch Row Navigation<\/b><\/h2>\n

                          Location:<\/b> Rivendale Farms<\/span><\/p>\n

                          Equipment:<\/b> Robot, 3 rows of plants<\/span><\/p>\n

                          Setup:<\/b><\/p>\n

                            \n
                          • Place robot at entrance to a row of plants, facing out of the row<\/span><\/li>\n
                          • Robot has pre-generated map file<\/span><\/li>\n<\/ul>\n

                            Test:<\/b><\/p>\n

                              \n
                            1. Power on the robot<\/span><\/li>\n
                            2. Establish connection to the robot<\/span><\/li>\n
                            3. Command the robot to switch rows<\/span><\/li>\n
                            4. Robot navigates to the beginning of the next row<\/span><\/li>\n
                            5. Robot stops at beginning of the row<\/span><\/li>\n<\/ol>\n

                              Success Criteria:<\/b><\/p>\n

                                \n
                              • Robot arrives at the entrance to the second row in at least 4 out of 5 trials (MR5)<\/span><\/li>\n
                              • Robot does not crush or trample any plants (MN5)<\/span><\/li>\n<\/ul>\n

                                 <\/p>\n

                                Test 3: Localization<\/b><\/h2>\n

                                Location:<\/b> Rivendale Farms<\/span><\/p>\n

                                Equipment:<\/b> Robot, pre-recorded validation ROS Bag, localization performance measurement node<\/span><\/p>\n

                                Setup:<\/b><\/p>\n

                                  \n
                                • Load pre-recorded ROS Bag file with ground truth (from RTK GPS) onto robot<\/span><\/li>\n<\/ul>\n

                                  Test:<\/b><\/p>\n

                                    \n
                                  1. Power on the robot<\/span><\/li>\n
                                  2. Establish connection to the robot<\/span><\/li>\n
                                  3. Start performance measurement node<\/span><\/li>\n
                                  4. Playback ROS Bag file and observe divergence of ground truth and the actual position<\/span><\/li>\n
                                  5. Observe output of localization validation node at end of the run<\/span><\/li>\n<\/ol>\n

                                    Success Criteria:<\/b><\/p>\n

                                      \n
                                    • Robot is in the correct row with 95% accuracy, and within 24 inches along the row (MR4)<\/span><\/li>\n<\/ul>\n

                                       <\/p>\n

                                      Test 4: Row Perception<\/b><\/h2>\n

                                      Location:<\/b> Rivendale Farms<\/span><\/p>\n

                                      Equipment:<\/b> Robot, pre-recorded validation ROS Bag, row perception performance measurement node<\/span><\/p>\n

                                      Setup:<\/b><\/p>\n

                                        \n
                                      • Load pre-recorded ROS Bag file with human-labeled ground truth<\/span><\/li>\n<\/ul>\n

                                        Test:<\/b><\/p>\n

                                          \n
                                        1. Power on the robot<\/span><\/li>\n
                                        2. Establish connection to the robot<\/span><\/li>\n
                                        3. Start performance measurement node<\/span><\/li>\n
                                        4. Playback ROS Bag file and observe divergence of ground truth and actual measurement<\/span><\/li>\n
                                        5. Observe output of row perception validation node at end of the run<\/span><\/li>\n<\/ol>\n

                                          Success Criteria:<\/b><\/p>\n