{"id":226,"date":"2019-04-06T01:57:01","date_gmt":"2019-04-06T01:57:01","guid":{"rendered":"http:\/\/mrsdprojects.ri.cmu.edu\/2018teame\/?page_id=226"},"modified":"2019-05-08T23:19:09","modified_gmt":"2019-05-08T23:19:09","slug":"spring-validation-demonstrations","status":"publish","type":"page","link":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teame\/system-performance\/spring-validation-demonstrations\/","title":{"rendered":"Spring Validation Demonstrations"},"content":{"rendered":"<h3><b>Test 1: Row Navigation<\/b><\/h3>\n<p><b>Location:<\/b><span style=\"font-weight: 400\"> Carnegie Mellon University, B Floor<\/span><\/p>\n<p><b>Equipment:<\/b><span style=\"font-weight: 400\"> Robot, 2 rows of artificial plants<\/span><\/p>\n<p><b>Setup:<\/b><\/p>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Place robot at the entrance of a row of artificial plants, facing into the row<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">The robot has a pre-generated map file<\/span><\/li>\n<\/ul>\n<p><b>Test:<\/b><\/p>\n<ol>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">The robot navigates to the end of the first row<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">The robot turns into the next row<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">The robot drives to the end of the second row<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Manually move the robot to start location and restart software; repeat 5 times<\/span><\/li>\n<\/ol>\n<p><b>Success Criteria:<\/b><\/p>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Robot fits in the row (MN1)<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Robot arrives and stops at the far end of rows 1 and 2<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">The robot does not crush or trample any artificial plants (MN5)<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">The robot successfully switches into the second row in at least 4 out of 5 trials (MR5)<\/span><\/li>\n<\/ul>\n<h3><b>Test 2: Localization<\/b><\/h3>\n<p><b>Location:<\/b><span style=\"font-weight: 400\"> Carnegie Mellon University<\/span><\/p>\n<p><b>Equipment:<\/b><span style=\"font-weight: 400\"> Robot, pre-recorded validation rosbag (software), localization performance measurement node (software)<\/span><\/p>\n<p><b>Setup:<\/b><\/p>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Load pre-recorded ROS Bag file with ground truth (from RTK GPS) onto the robot<\/span><\/li>\n<\/ul>\n<p><b>Test:<\/b><\/p>\n<ol>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Playback ROS Bag file and observe divergence of ground truth and the actual position<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Observe the output of the localization validation node at the end of the run<\/span><\/li>\n<\/ol>\n<p><b>Success Criteria:<\/b><\/p>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">The robot is in the correct row with 80% accuracy, and within 24 inches along the row (MR4)<\/span><\/li>\n<\/ul>\n<h3><b>Test 3: Pest\/Disease Perception Software Test<\/b><\/h3>\n<p><b>Location:<\/b><span style=\"font-weight: 400\"> Carnegie Mellon University<\/span><\/p>\n<p><b>Equipment:<\/b><span style=\"font-weight: 400\"> Robot, pre-collected and labeled dataset<\/span><\/p>\n<p><b>Test ( Video Demo)<\/b><\/p>\n<ol>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">The monitor software runs on test images of 1 type of plant and predicts severity* of holes and fungus for each image base on their leave area, hole area, and fungus area <\/span><\/li>\n<\/ol>\n<p><b>Success Criteria<\/b><\/p>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\"> \u00a0successfully identifies fungus and holes severity with greater than 50% micro precision and recall* (MR9, MR10)<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">The robot successfully processes data at a rate faster than one field per 24 hours (MR 12)<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400\">*severity levels: mild,, moderate and alarming, represented as integers 1,2 and 3<\/span><\/p>\n<p><span style=\"font-weight: 400\">*micro precision: all TP \u00a0\/ all TP + all FP<\/span><\/p>\n<p><span style=\"font-weight: 400\">*micro recall<\/span> <span style=\"font-weight: 400\">: all TP \u00a0\/ all TP + all FN<\/span><\/p>\n<p>&nbsp;<\/p>\n<h3>Discussion<\/h3>\n<p><span style=\"font-weight: 400\">The SVD and SVD Encore tested the robot\u2019s ability to traverse the field, accurately localize itself and evaluate the severity of pest and disease pressures. <\/span><\/p>\n<p><span style=\"font-weight: 400\">The first test validated the system\u2019s ability to fit in a row of 24 in, autonomously switch rows 80% of the times and not damage plants during navigation. The system was expected to traverse a row, switch to the next one and then traverse till the end of the next row. The system passed the test during SVD. However, while turning, the robot crushed the plant at the end of the row. This was because the row-detector could not detect the row until the robot was halfway into the row. Thus it completely relied on dead reckoning using visual odometry which was drifted and thus led to the collision. In our SVD encore, the robot, unfortunately, collided with plants while entering the second row on two occasions. This error seemed to mostly stem from dead reckoning as well, as the robot attempted to enter the second row in the wrong position, before actually seeing the row. The discrepancy in the location estimate between the two SVDs in unclear, however, possible sources are tweaks to noise parameters in the motion model and a ZED sensor mount which came slightly loose.<\/span><\/p>\n<p><span style=\"font-weight: 400\">The second test was to validate the robots ability to localize itself within the correct row 80% of the times and along the row within 24in. The system was provided with sensor data and ground truth in the form of RTK GPS data and was expected to localize within error bounds. In the first SVD, the localizer was not able to satisfy the required error bounds. This was used using a dataset with wheel odometry only, as we hadn\u2019t visited Rivendale Farms to collect a dataset since we mounted the ZED sensor which provides visual odometry. Our Rivendale test was rained out, so we instead collected a dataset at CMU with visual odometry. During SVD encore, the robot passed the test within the error bounds specified.<\/span><\/p>\n<p><span style=\"font-weight: 400\">The third tested validated the robot\u2019s ability to identify disease and holes. The robot was provided with images of plants with disease and holes. The trained neural network was able to categorize the image into levels of severity. The system was expected to categorize the holes and weeds with 50% of precision and recall and was able to achieve ~70% precision and recall.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Test 1: Row Navigation Location: Carnegie Mellon University, B Floor Equipment: Robot, 2 rows of artificial plants Setup: Place robot at the entrance of a row of artificial plants, facing into the row The robot has a pre-generated map file Test: The robot navigates to the end of the first row The robot turns into &hellip; <a href=\"https:\/\/mrsdprojects.ri.cmu.edu\/2018teame\/system-performance\/spring-validation-demonstrations\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Spring Validation Demonstrations&#8221;<\/span><\/a><\/p>\n","protected":false},"author":158,"featured_media":0,"parent":41,"menu_order":1,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-226","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teame\/wp-json\/wp\/v2\/pages\/226","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teame\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teame\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teame\/wp-json\/wp\/v2\/users\/158"}],"replies":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teame\/wp-json\/wp\/v2\/comments?post=226"}],"version-history":[{"count":5,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teame\/wp-json\/wp\/v2\/pages\/226\/revisions"}],"predecessor-version":[{"id":311,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teame\/wp-json\/wp\/v2\/pages\/226\/revisions\/311"}],"up":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teame\/wp-json\/wp\/v2\/pages\/41"}],"wp:attachment":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teame\/wp-json\/wp\/v2\/media?parent=226"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}