Test Plan

Fall Validation Experiments

We have eight validation tests planned in Fall 2019, these include sub-system level tests for monitoring and navigation sub-systems and complete system-level tests as well.

 

Progress Review Capability Milestones Associated Tests Associated Requirements
#8 (9/25)
  • Visualizer V1 complete
  • RTK Node complete
#9 (10/9)
  • MVP of Monitoring complete
  • Localization integrated
  • Camera exposure v1 complete
 Test 05 MN1,2
#10 (10/23)
  • Navigation MVP complete
 Test 03, 04, 07 MR 3,4,5,6 (MR 6.1 – 6.4)
#11 (11/6)
  • Wheel covers installed
 Test 01, 02, 06 MR 1, MR 2

MN 4
#12 (11/18)
  • Integration and Testing
FVD (11/25)
  • Full system integration
 Test 08 MR 1-6

MN 1-4

Note: Generally, tests are scheduled for the PR after the capability milestone has been achieved to leave room for finishing touches and dry runs before presenting the results. 

 

Tests

 

Test Number: 01 Test Name:

Autonomous Navigation Test

 Test Date: 11/6
Objective: Autonomously traverse field without hitting plant stem (MN4, MR1, MR2)
Elements Tested: Subsystem: Autonomous Row Navigation
Location: Rivendale Farms – Recorded on Video
Equipment:
  • See field visit checklist
  • Map of the field
  • Cameras
 Personnel:
  • Team E
  • Rivendale farm representatives (Not required)
Procedure:
  1. Place the robot at the start of the first row
  2. Command the robot to start autonomous navigation
  3. Robot traverses the row and switches to the next row
  4. The robot stops after traversing 5 rows
Verification Criteria:
  • Robot does not hit plant stems while navigating
  • Robot successfully switches row 4 out of 5 times

 

Test Number: 02 Test Name:

Coverage Planner Test

 Test Date: 11/6
Objective: Verify coverage plan covers the entire field of interest 
Elements Tested: Subsystem: Coverage Planner
Location: Newell Simon Hall
Equipment:
  • Map File
  • Computer
 Personnel:
  • Team E
  • MRSD Advisors
Procedure:
  1. Load the map file
  2. Software generates the coverage plan
  3. Play animation showing the entire coverage plan
Verification Criteria:
  • Coverage planner covers the rows of interest that the robot can reach

 

Test Number: 03 Test Name:

Pest/Disease Perception Software Test

 Test Date: 10/23
Objective: Evaluate the performance of plant health monitoring deep net (MR4, MR5)
Elements Tested: Subsystem: Mask-RCNN (Perception)
Location: Newell Simon Hall
Equipment:
  • Server with GPU
  • Test images (labeled {positive, negative}) from a new spatial area of the field, on which we have not trained before
 Personnel:
  • Team E
  • MRSD Advisors
Procedure:
  1. Power on gpu server
  2. Run plant health model inference
  3. Software loads test images and computes performance metric
Verification Criteria:
  • Robot successfully identifies fungus and holes

with greater than 80% precision and recall with unhealthy defined as positive 

            * precision = Tp / (Fp + Tp), recall = Tp / (Fn + Tp)

 

Test Number: 04 Test Name: Visualization Subsystem Test Test Date: 10/23
Objective: Evaluate the speed of monitoring pipeline and GUI Features (MR6)
Elements Tested: Subsystem: Monitoring pipeline 
Location: Newell Simon Hall
Equipment:
  • GPU Server 
  • Pre-recorded ROS bag files of the entire rows of interest
 Personnel:
  • Team E
  • MRSD Advisors
Procedure:
  1. Power on GPU server
  2. Run visualization pipeline
  3. Software loads test images and visualizes the results in GUI
  4. Visually demonstrate the field’s layout
  5. Select a datapoint and change the classification, to demonstrate change in the stored data
  6. End the visualizer and relaunch it to show the same data
Verification Criteria:
  • Robot successfully processes data at a rate faster

than one field per 24 hours (MR 6)

  • Clear depiction of the field layout and data presentation (subjective)
  • Successful use of interactive portions of the GUI
  • Successful data preservation on relaunch

Note: The GUI features are subject to change, pending product reviews with the farmers

 

Test Number: 05 Test Name:

Robot Platform Verification

 Test Date: 10/9
Objective: Verify Non-Functional Requirements related to the Robot Platform 

(MN1, MN2)
Elements Tested: Subsystem: Robot Platform
Location: Rivendale Farms
Equipment:
  • Video Camera
  • Robot
  • Tape Measure
 Personnel:
  • Team E
  • MRSD Advisors
Procedure:
  1. Power on robot
  2. Drive the robot via joystick
  3. Continue to press on joystick, releasing the remote kill switch, demonstrating a stop
  4. Drive the robot via joystick
  5. Press the mechanical E-Stop to demonstrate a stop
  6. Drive the robot to the beginning of the first row of the brassica field
  7. Measure the open space between the robot and the plant stems, to demonstrate plant clearance
Verification Criteria:
  • Effectiveness of remote kill switch
  • Effectiveness of mechanical E-stop
  • Fit of robot into field’s rows

 

Test Number: 06 Test Name:

Battery Life Test

 Test Date: 11/6
Objective: Confirm battery life is sufficient for the Rivendale Brassica Field (MN3)
Elements Tested: Subsystem: Robot Platform
Location: Rivendale Farms
Equipment:
  • Robot
  • Video camera
 Personnel:
  • Team E
  • MRSD Advisors
Procedure:
  1. Place the fully charged robot at the start of the first row
  2. Drive the robot at standard speed, through the entire brassica field
Verification Criteria:
  • Sufficient Battery Life to cover the entire brassica field

 

Test Number: 07 Test Name:

Usable row images

 Test Date: 10/27
Objective: Confirm that we are collecting images of acceptable exposure which are usable for the deep learning pipeline. (MR3)
Elements Tested: Subsystem: Mask-RCNN (Perception)
Location: Newell Simon Hall
Equipment:
  • ROS Bag
  • ROS-enabled laptop
 Personnel:
  • Team E
  • MRSD Advisors
Procedure:
  1. Take a ROS Bag consisting of left and right camera images collected from a row traversal as input. 
  2. Use the exposure testing script to process images and find the percentage of images that pass an over/underexposed test. 
Verification Criteria:
  • The percentage of images that pass the test should be > 75% 

 

Test Number: 08 Test Name:

System Integration Test

 Test Date: 11/25
Objective: Verify end-to-end robot system functionality ( MR 1-6, MN1-4)
Elements Tested: Robot Platform, Navigation, Plant Health Monitoring, GUI
Location: Rivendale Farms / Newell-Simon Hall
Equipment: Rivendale Farms

  • Video Camera
  • Robot
  • Joystick
  • Portable hard drive

Newell-Simon Hall

  • Server with GPU
 Personnel:
  • Team E (Rivendale Farms)
  • MRSD Advisors (Newell-Simon Hall)
Procedure:
  1. Set up video camera and begin recording
  2. Complete Test 01 Autonomous Navigation Test
  3. End recording of video
  4. Copy the ROS bag from the plant health monitoring run to the portable hard drive
  5. Transport the hard drive to the MRSD lab and copy files to server for inference
  6. Run Test 07 Usable Row Images Test on server
  7. Run Test 03 Pest/Disease Perception Software Test
  8. Complete Test 04 GUI Feature Verification for this new data
Verification Criteria:
  • Video correctly shows criteria for Test 01, Autonomous Navigation passed
  • Collected data passes Test 07 Usable Row Images test
  • System passes Test 03 Pest/Disease Perception Software Test on newly collected data
  • System passes Test 04 GUI Feature Verification on newly collected data

Spring Validation Experiments

We have planned five validation tests in the spring of 2019 that are focused primarily on navigation. As such, their success criteria correspond to the navigation-related performance requirements. We have also tried to minimize the number of in-field tests required, to save on time and resources, so tests 3 through 5, which are primarily software systems, will be tested on pre-recorded data.

Test 1: In-Row Navigation

Location: Rivendale Farms

Equipment: Robot, 2 rows of plants

Setup:

  • Place robot at the entrance to a row of plants, facing into the row
  • Robot has pre-generated map file

Test:

  1. Power on the robot
  2. Establish connection to the robot
  3. Command the robot to traverse the row
  4. Robot navigates along the row
  5. Robot stops at end of row

Success Criteria:

  • Robot fits in row (MN1)
  • Robot arrives at the far end of row
  • Robot does not crush or trample any plants (MN5)

 

Test 2: Switch Row Navigation

Location: Rivendale Farms

Equipment: Robot, 3 rows of plants

Setup:

  • Place robot at entrance to a row of plants, facing out of the row
  • Robot has pre-generated map file

Test:

  1. Power on the robot
  2. Establish connection to the robot
  3. Command the robot to switch rows
  4. Robot navigates to the beginning of the next row
  5. Robot stops at beginning of the row

Success Criteria:

  • Robot arrives at the entrance to the second row in at least 4 out of 5 trials (MR5)
  • Robot does not crush or trample any plants (MN5)

 

Test 3: Localization

Location: Rivendale Farms

Equipment: Robot, pre-recorded validation ROS Bag, localization performance measurement node

Setup:

  • Load pre-recorded ROS Bag file with ground truth (from RTK GPS) onto robot

Test:

  1. Power on the robot
  2. Establish connection to the robot
  3. Start performance measurement node
  4. Playback ROS Bag file and observe divergence of ground truth and the actual position
  5. Observe output of localization validation node at end of the run

Success Criteria:

  • Robot is in the correct row with 95% accuracy, and within 24 inches along the row (MR4)

 

Test 4: Row Perception

Location: Rivendale Farms

Equipment: Robot, pre-recorded validation ROS Bag, row perception performance measurement node

Setup:

  • Load pre-recorded ROS Bag file with human-labeled ground truth

Test:

  1. Power on the robot
  2. Establish connection to the robot
  3. Start performance measurement node
  4. Playback ROS Bag file and observe divergence of ground truth and actual measurement
  5. Observe output of row perception validation node at end of the run

Success Criteria:

  • Robot perceives drivable width of row within -10% error bound (MR3)

 

Test 5: Mapping Accuracy

Location: Rivendale Farms

Equipment: Robot, pre-recorded sensor data of full field traversal, manually generated map

Setup:

  • Load pre-recorded ROS Bag file with human-labeled ground truth

Test:

  1. Power on the robot
  2. Establish connection to the robot
  3. Robot generates a map using pre-recorded sensor data of full field traversal
  4. Compare known location of visual markers with those of generated map

Success Criteria:

  • The map has a maximum 15% dimensional error (MR2)

 

Fall Validation Experiments

Test 1: Pest/Disease Perception Test

Location: Rivendale Farms

Equipment: Robot, pre-collected and labeled dataset

Test

  1. Power on the robot
  2. Establish a connection to the robot
  3. Robot processes images and delivers a report on the number and location of plant problems (which problems will be decided later)
  4. Robot report compared to the labeled dataset

Success Criteria

  • Robot successfully identifies problems with less than 20% false positives or false negatives (MR9, MR10)
  • Robot successfully processes data at a rate faster than one field per 24 hours (MR 12)

 

Test 2: Weeding Perception Test

Location: Rivendale Farms

Equipment: Robot, pre-collected and labeled dataset

Test

  1. Power on the robot
  2. Establish a connection to the robot
  3. Robot processes images and delivers a report on the number and location of plant problems (which problems will be decided later)
  4. Robot report compared to the labeled dataset

Success Criteria

  • Robot successfully identifies weeds with false positive on plant < 5%, false negative < 30% (MR7)
  • Robot successfully localizes identified weeds to the positional error of <2” with respect to the robot’s frame (MR8)
  • Robot successfully processes data at a speed allowing for full coverage of field at robot’s weeding mode speed (MR7, MR8)

 

Test 3: Mechanical Weeding Test

Location: Rivendale Farms

Equipment: Robot, a bed of plants with weeds present, labeled data for weed locations

Setup:

  • Place robot at plant bed, with weeding manipulator facing the bed

Test

  1. Power on the robot
  2. Establish a connection to the robot
  3. Robot records plant images
  4. Robot processes data online and actuates the mechanical weeder

Success Criteria

  • Robot successfully removes 75% of weeds, by coverage area (MR 11)
  • The robot does not damage the plant (MN 6)

 

Test 4: Monitoring Systems-level Test

Location: Rivendale Farms

Equipment: Robot, map file, brassica field

Setup:

  • Place robot at the start of field

Test

  1. Power on the robot
  2. Establish a connection to the robot
  3. Robot autonomously navigates and localizes
  4. Robot captures images of plants
  5. Robot returns to the starting point
  6. Robot process images
  7. Robot generates and sends a report

Success Criteria

  • The robot does not damage plants (MN 5)
  • Robot generates a report in under 24 hours from completion of the test (MR12)

 

Test 5: Weeding Systems-level Test

Location: Rivendale Farms

Equipment: Robot, map file, 1 row of plants, human captured pictures of weeds in row

Setup:

  • Place robot at the start of field

Test

  1. Power on the robot
  2. Establish a connection to the robot
  3. Robot autonomously navigates and localizes along 1 row
  4. Robot captures images of plants
  5. Robot process images
  6. Robot Mechanically weeds field
  7. Robot returns to the starting point
  8. Robot generates and sends a report
  9. Human captured after pictures for the row are compared to before pictures

Success Criteria

  • The robot does not damage plants (MN 6)
  • The robot removes at least 75% of weeds by coverage area (MR 11)
  • Robot generates a report in under 24 hours from completion of the test (MR12)