{"id":228,"date":"2019-04-06T01:59:50","date_gmt":"2019-04-06T01:59:50","guid":{"rendered":"http:\/\/mrsdprojects.ri.cmu.edu\/2018teame\/?page_id=228"},"modified":"2019-12-13T19:06:50","modified_gmt":"2019-12-13T19:06:50","slug":"fall-validation-demonstrations","status":"publish","type":"page","link":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teame\/fall-validation-demonstrations\/","title":{"rendered":"Fall Validation Demonstrations"},"content":{"rendered":"<p>&nbsp;<\/p>\n<table>\n<tbody>\n<tr>\n<td><b>Test Name:<\/b><\/td>\n<td colspan=\"4\">\n<h2><span style=\"font-weight: 400\">Fall Validation Demonstration<\/span><\/h2>\n<\/td>\n<td><b>Test Date:<\/b><\/td>\n<td><span style=\"font-weight: 400\">11\/25<\/span><\/td>\n<\/tr>\n<tr>\n<td><b>Objective:<\/b><\/td>\n<td colspan=\"6\"><span style=\"font-weight: 400\">Verify end-to-end robot system functionality<\/span><\/td>\n<\/tr>\n<tr>\n<td><b>Elements Tested:<\/b><\/td>\n<td colspan=\"6\"><span style=\"font-weight: 400\">Robot Platform, Navigation, Plant Health Monitoring, GUI<\/span><\/td>\n<\/tr>\n<tr>\n<td><b>Location:<\/b><\/td>\n<td colspan=\"6\"><span style=\"font-weight: 400\">Newell-Simon Hall<\/span><\/td>\n<\/tr>\n<tr>\n<td><b>Equipment:<\/b><\/td>\n<td colspan=\"3\"><span style=\"font-weight: 400\">Newell-Simon Hall<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Server with GPU<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">ROS Bag from field visit<\/span><\/li>\n<\/ul>\n<\/td>\n<td><b>Personnel:<\/b><\/td>\n<td colspan=\"2\">\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Team E\u00a0<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">MRSD Advisors<\/span><\/li>\n<\/ul>\n<\/td>\n<\/tr>\n<tr>\n<td><b>Procedure:<\/b><\/td>\n<td colspan=\"6\">\n<ol>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">View the video of the Rivendale field test<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Use the exposure testing script to process the ROS Bag and find the percentage of images which pass an over\/under exposed test.\u00a0<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Run plant health model inference on ROS Bag<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Software loads test images and computes performance metric<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Run visualization pipeline<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Software loads test images and visualizes the results in GUI<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Visually demonstrate the field\u2019s layout<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Select a datapoint and change the classification, to demonstrate change in the stored data<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">End the visualizer and relaunch it to show the same data<\/span><\/li>\n<\/ol>\n<\/td>\n<\/tr>\n<tr>\n<td><b>Verification Criteria:<\/b><\/td>\n<td colspan=\"6\">\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Video correctly shows criteria for Test 1, Autonomous Navigation passed<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">The percentage of images that pass the exposure test should be &gt; 75%<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Robot successfully processes data at a rate faster<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400\">than one field per 24 hours (MR 11)<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Robot successfully identifies fungus and holes<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400\">with greater than 80% precision and recall with unhealthy defined as positive\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0* precision = Tp \/ (Fp + Tp), recall = Tp \/ (Fn + Tp)<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Clear depiction of the field layout and data presentation (subjective)<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Successful use of interactive portions of the GUI<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Successful data preservation on relaunch<\/span><\/li>\n<\/ul>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>&nbsp;<\/p>\n<p><strong>Results<\/strong><\/p>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Navigation<\/span><\/li>\n<\/ul>\n<ul>\n<li style=\"list-style-type: none\">\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Robot does not hit plant stems while navigating &#8211; <a href=\"https:\/\/www.youtube.com\/watch?v=PJAyZ2VIdIM&amp;feature=emb_logo\">view here<\/a><\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Robot traverses till the end of the first row &#8211; <a href=\"https:\/\/www.youtube.com\/watch?v=PJAyZ2VIdIM&amp;feature=emb_logo\">view here<\/a><\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Robot successfully switches between two navigable rows 4 out of 5 times &#8211; <a href=\"https:\/\/www.youtube.com\/watch?v=8ASy9B0D3pA&amp;feature=emb_logo\">view here<\/a><\/span><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Monitoring\u00a0<\/span><\/li>\n<\/ul>\n<ul>\n<li style=\"list-style-type: none\">\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Robot successfully processes data at a rate faster than one field per 24 hours (MR 11)\u00a0 &#8211; 11.25 hours max <a href=\"https:\/\/youtu.be\/E34IkUxelmQ\">view here<\/a><\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Robot successfully identifies fungus with greater than 80% precision and recall &#8211; see below<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Robot successfully identifies holes with greater than 80% precision and recall &#8211; see below<\/span><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<table>\n<tbody>\n<tr>\n<td colspan=\"4\">\n<p style=\"text-align: center\"><strong>Binary Classification Performance<\/strong><\/p>\n<\/td>\n<\/tr>\n<tr>\n<td><strong>Plant Type<\/strong><\/td>\n<td><strong>Category<\/strong><\/td>\n<td><strong>Precision<\/strong><\/td>\n<td><strong>Recall<\/strong><\/td>\n<\/tr>\n<tr>\n<td rowspan=\"2\"><span style=\"font-weight: 400\">Broccolini<\/span><\/td>\n<td><span style=\"font-weight: 400\">Pest Hole<\/span><\/td>\n<td><span style=\"font-weight: 400\">79%<\/span><\/td>\n<td><span style=\"font-weight: 400\">100%<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400\">Fungus<\/span><\/td>\n<td><span style=\"font-weight: 400\">95%<\/span><\/td>\n<td><span style=\"font-weight: 400\">95%<\/span><\/td>\n<\/tr>\n<tr>\n<td rowspan=\"2\"><span style=\"font-weight: 400\">Cabbage<\/span><\/td>\n<td><span style=\"font-weight: 400\">Pest Hole<\/span><\/td>\n<td><span style=\"font-weight: 400\">78%<\/span><\/td>\n<td><span style=\"font-weight: 400\">64%<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400\">Fungus<\/span><\/td>\n<td><span style=\"font-weight: 400\">100%<\/span><\/td>\n<td><span style=\"font-weight: 400\">91%<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400\">Curly Kale<\/span><\/td>\n<td><span style=\"font-weight: 400\">Fungus<\/span><\/td>\n<td><span style=\"font-weight: 400\">91%<\/span><\/td>\n<td><span style=\"font-weight: 400\">83%<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">User Interface<\/span><\/li>\n<\/ul>\n<ul>\n<li style=\"list-style-type: none\">\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Clear depiction of the field layout and data presentation (subjective) &#8211; cleared with farmers<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Successful use of interactive portions of the GUI &#8211; demonstrated live<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Successful data preservation on relaunch of GUI &#8211; demonstrated live<\/span><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>&nbsp; Test Name: Fall Validation Demonstration Test Date: 11\/25 Objective: Verify end-to-end robot system functionality Elements Tested: Robot Platform, Navigation, Plant Health Monitoring, GUI Location: Newell-Simon Hall Equipment: Newell-Simon Hall Server with GPU ROS Bag from field visit Personnel: Team E\u00a0 MRSD Advisors Procedure: View the video of the Rivendale field test Use the exposure &hellip; <a href=\"https:\/\/mrsdprojects.ri.cmu.edu\/2018teame\/fall-validation-demonstrations\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Fall Validation Demonstrations&#8221;<\/span><\/a><\/p>\n","protected":false},"author":158,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-228","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teame\/wp-json\/wp\/v2\/pages\/228","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teame\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teame\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teame\/wp-json\/wp\/v2\/users\/158"}],"replies":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teame\/wp-json\/wp\/v2\/comments?post=228"}],"version-history":[{"count":8,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teame\/wp-json\/wp\/v2\/pages\/228\/revisions"}],"predecessor-version":[{"id":497,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teame\/wp-json\/wp\/v2\/pages\/228\/revisions\/497"}],"wp:attachment":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teame\/wp-json\/wp\/v2\/media?parent=228"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}