{"id":64,"date":"2020-01-15T04:21:34","date_gmt":"2020-01-15T04:21:34","guid":{"rendered":"http:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/?page_id=64"},"modified":"2020-12-19T00:08:50","modified_gmt":"2020-12-19T00:08:50","slug":"system-performance","status":"publish","type":"page","link":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/system-performance\/","title":{"rendered":"System Performance"},"content":{"rendered":"<h2>Fall Validation Demonstration Results<\/h2>\n<p>We performed two sets of full system tests for our Fall Validation Demos; one in simulation, and one on the Husky platform. The desired requirements for both tests are listed below.<\/p>\n<ul>\n<li>Final translational error &lt; 75 cm<\/li>\n<li>Final rotational error &lt; 10 deg<\/li>\n<li>Total runtime &lt; 2 x tele-op runtime<\/li>\n<\/ul>\n<h3>a) <span style=\"font-size: inherit\">Simulation Results:<\/span><\/h3>\n<p><span style=\"font-size: inherit\"> To fully validate system robustness, we created four Gazebo worlds. These four worlds cover all types of scenarios we expect our system to be able to handle for parking lot traversal (making right\/left turns around the bend) and parking (parking on the right\/left side of the road). For each world, we performed three autonomous runs (one on each of our laptops simultaneously) and compared their results with one manually teleoperated run (on Nikita\u2019s laptop). The two tables below show our final results for the world in which the Husky makes two right turns around the bend and parks in a parking spot to its right side.<\/span><\/p>\n\n<table id=\"tablepress-13\" class=\"tablepress tablepress-id-13\">\n<thead>\n<tr class=\"row-1\">\n\t<th class=\"column-1\">RRR World<\/th><th class=\"column-2\">Nikita (Tele-operator)<\/th><th class=\"column-3\">Subbu (Machine 1)<\/th><th class=\"column-4\">Eva (Machine 2)<\/th><th class=\"column-5\">Kevin (Machine 3)<\/th><th class=\"column-6\">Average<\/th>\n<\/tr>\n<\/thead>\n<tbody class=\"row-striping row-hover\">\n<tr class=\"row-2\">\n\t<td class=\"column-1\">Time (s)<\/td><td class=\"column-2\">77<\/td><td class=\"column-3\">86<\/td><td class=\"column-4\">105<\/td><td class=\"column-5\">122<\/td><td class=\"column-6\">104.33<\/td>\n<\/tr>\n<tr class=\"row-3\">\n\t<td class=\"column-1\">Parking Translational error (m)<\/td><td class=\"column-2\">N\/A<\/td><td class=\"column-3\">50.03<\/td><td class=\"column-4\">60.20<\/td><td class=\"column-5\">78.95<\/td><td class=\"column-6\">63.06<\/td>\n<\/tr>\n<tr class=\"row-4\">\n\t<td class=\"column-1\">Parking Rotational error (deg)<\/td><td class=\"column-2\">N\/A<\/td><td class=\"column-3\">20.3<\/td><td class=\"column-4\">20.70<\/td><td class=\"column-5\">16.27<\/td><td class=\"column-6\">19.09<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<!-- #tablepress-13 from cache -->\n\n<table id=\"tablepress-14\" class=\"tablepress tablepress-id-14\">\n<thead>\n<tr class=\"row-1\">\n\t<th class=\"column-1\">Success Criteria<\/th><th class=\"column-2\">Results<\/th><th class=\"column-3\">% Success Rate<\/th><th class=\"column-4\">Avg. deviation from desired value<\/th>\n<\/tr>\n<\/thead>\n<tbody class=\"row-striping row-hover\">\n<tr class=\"row-2\">\n\t<td class=\"column-1\">Time &lt; 154s<\/td><td class=\"column-2\">PASS<\/td><td class=\"column-3\">100%<\/td><td class=\"column-4\">0<\/td>\n<\/tr>\n<tr class=\"row-3\">\n\t<td class=\"column-1\">Translational error &lt; 75%<\/td><td class=\"column-2\">PASS<\/td><td class=\"column-3\">66.66%<\/td><td class=\"column-4\">0<\/td>\n<\/tr>\n<tr class=\"row-4\">\n\t<td class=\"column-1\">Rotational error &lt; 10 deg<\/td><td class=\"column-2\">FAIL<\/td><td class=\"column-3\">0%<\/td><td class=\"column-4\">9.09 degrees<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<!-- #tablepress-14 from cache -->\n<p><span style=\"font-size: inherit\">The tables below show our final results for the world in which the Husky takes two left turns around the bend and parks in a parking spot to its left side.<\/span><\/p>\n\n<table id=\"tablepress-15\" class=\"tablepress tablepress-id-15\">\n<thead>\n<tr class=\"row-1\">\n\t<th class=\"column-1\">LLL World<\/th><th class=\"column-2\">Nikita (Manual)<\/th><th class=\"column-3\">Subbu (Machine 1)<\/th><th class=\"column-4\">Eva (Machine 2)<\/th><th class=\"column-5\">Kevin (Machine 3)<\/th><th class=\"column-6\">Average<\/th>\n<\/tr>\n<\/thead>\n<tbody class=\"row-striping row-hover\">\n<tr class=\"row-2\">\n\t<td class=\"column-1\">Time(s)<\/td><td class=\"column-2\">94<\/td><td class=\"column-3\">87<\/td><td class=\"column-4\">85<\/td><td class=\"column-5\">94<\/td><td class=\"column-6\">88.67<\/td>\n<\/tr>\n<tr class=\"row-3\">\n\t<td class=\"column-1\">Parking translational error (cm)<\/td><td class=\"column-2\">N\/A<\/td><td class=\"column-3\">34.45<\/td><td class=\"column-4\">13.85<\/td><td class=\"column-5\">20.19<\/td><td class=\"column-6\">22.83<\/td>\n<\/tr>\n<tr class=\"row-4\">\n\t<td class=\"column-1\">Parking rotational error (deg)<\/td><td class=\"column-2\">N\/A<\/td><td class=\"column-3\">0.24<\/td><td class=\"column-4\">11.85<\/td><td class=\"column-5\">0.84<\/td><td class=\"column-6\">4.31<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<!-- #tablepress-15 from cache -->\n\n<table id=\"tablepress-16\" class=\"tablepress tablepress-id-16\">\n<thead>\n<tr class=\"row-1\">\n\t<th class=\"column-1\">Success Criteria<\/th><th class=\"column-2\">Results<\/th><th class=\"column-3\">% Success Rate<\/th><th class=\"column-4\">Avg. deviation from desired value<\/th>\n<\/tr>\n<\/thead>\n<tbody class=\"row-striping row-hover\">\n<tr class=\"row-2\">\n\t<td class=\"column-1\">Time &lt; 188s<\/td><td class=\"column-2\">PASS<\/td><td class=\"column-3\">100%<\/td><td class=\"column-4\">0<\/td>\n<\/tr>\n<tr class=\"row-3\">\n\t<td class=\"column-1\">Translational error &lt; 75 cms<\/td><td class=\"column-2\">PASS<\/td><td class=\"column-3\">100%<\/td><td class=\"column-4\">0<\/td>\n<\/tr>\n<tr class=\"row-4\">\n\t<td class=\"column-1\">Rotational error &lt; 10 degrees<\/td><td class=\"column-2\">PASS<\/td><td class=\"column-3\">`66.66%<\/td><td class=\"column-4\">1.85 degrees<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<!-- #tablepress-16 from cache -->\n<p>Inference: All 6 runs in simulation passed the timing criteria and 2 out of 6 runs were fully successful in all aspects. There were 5 out of 6 successful runs with respect to translational errors, and 2 out of 6 successful runs with respect to orientational errors. Overall, the unsuccessful runs were only off by an average of 3.95 cm in translation and 7.28 degrees in orientation. <a href=\"https:\/\/drive.google.com\/file\/d\/1kh_mM-6rMkuWWeuVhRDicaP-okp2A1_c\/view?usp=sharing\">Here<\/a> is a link to one of our test runs in simulation.<\/p>\n<h3>b) Hardware results:<\/h3>\n<p>We performed tests to validate full system functionality for two scenarios\u2013two right turns and parking on the right side, and two right turns and parking on the left side. At the end of our demonstration, we asked the instructors to select a random parking spot, and the Husky was able to successfully park itself within the requirements. <a href=\"https:\/\/drive.google.com\/file\/d\/1Tql3jo871WrZIlvU_maizwaCCL0-xV7V\/view?usp=sharing\">Here<\/a> is a link to one of our test runs on hardware.<\/p>\n<p>1. Two right turns and right-side parking &#8211;<\/p>\n<ul>\n<li>RESULTS: Time = 158s;<\/li>\n<li>Translation error = 66cm;<\/li>\n<li>Rotation error = 2.1 degrees<\/li>\n<li>INFERENCE: Mostly successful. During this run, the robot froze on one of the turns, we had to give it a slight &#8220;nudge&#8221; with the joystick, and then it got back on track and finished autonomous functionality.<\/li>\n<\/ul>\n<p>2. Two right turns and left-side parking &#8211;<\/p>\n<ul>\n<li>RESULTS: Time = 115s;<\/li>\n<li>Translation error = 50cm;<\/li>\n<li>Rotation error = 8.6 degrees<\/li>\n<li>INFERENCE: Successful full run within all requirements.<\/li>\n<\/ul>\n<p>3. Two right turns and right-side parking &#8211;<\/p>\n<ul>\n<li>RESULTS: Time = 101s;<\/li>\n<li>Translation error = 95cm;<\/li>\n<li>Rotation error = 5.8 degrees<\/li>\n<li>INFERENCE: Successful full run, but translation error above requirement by 20 cm<\/li>\n<\/ul>\n<p>4. Random parking spot selection &#8211;<\/p>\n<ul>\n<li>RESULTS: Time = 66s;<\/li>\n<li>Translation error = 38cm;<\/li>\n<li>Rotation error = 5.1 degrees<\/li>\n<li>INFERENCE: Successful full run within all requirements<\/li>\n<\/ul>\n<h2 id=\"tablepress-10-name\" class=\"tablepress-table-name tablepress-table-name-id-10\">Spring Validation Demonstration Results<\/h2>\n\n<table id=\"tablepress-10\" class=\"tablepress tablepress-id-10\" aria-labelledby=\"tablepress-10-name\">\n<thead>\n<tr class=\"row-1\">\n\t<th class=\"column-1\">Requirement<\/th><th class=\"column-2\">Description<\/th><th class=\"column-3\">Achieved<\/th>\n<\/tr>\n<\/thead>\n<tbody class=\"row-striping row-hover\">\n<tr class=\"row-2\">\n\t<td class=\"column-1\">M.P.2<\/td><td class=\"column-2\">System will create a map of the environment with a mapping error of less than 20 cm as compared to the ground truth map<\/td><td class=\"column-3\">16.305 cm<\/td>\n<\/tr>\n<tr class=\"row-3\">\n\t<td class=\"column-1\">M.P.3<\/td><td class=\"column-2\">System\u2019s map building will be robust over sharp turns, such that the SLAM pipeline will not break (lose track of correspondences) in 80% of trials where a turn is present<\/td><td class=\"column-3\">100%<\/td>\n<\/tr>\n<tr class=\"row-4\">\n\t<td class=\"column-1\">M.P.4<\/td><td class=\"column-2\">System will discretely localize itself within 5 degrees in rotation and 50 cm in translation as compared to the ground truth location<\/td><td class=\"column-3\">48.836 cm translation,<br \/>\n2.099 deg rotation<\/td>\n<\/tr>\n<tr class=\"row-5\">\n\t<td class=\"column-1\">M.P.5<\/td><td class=\"column-2\">System will detect lanes in all frames with an Intersection over Union (IOU) score of greater than 60%<\/td><td class=\"column-3\">98%<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<!-- #tablepress-10 from cache -->\n<h2><\/h2>\n<h2><strong>SVD Test 1 (Mapping Test)<\/strong><\/h2>\n<ul>\n<li>We placed several unit boxes with known locations in our custom Gazebo environment.<\/li>\n<li>The Husky was tele-operated around the parking lot, creating a map of the world<\/li>\n<li>We ran an evaluation script where the user selects the map points corresponding to the corners of the unit boxes.<\/li>\n<li>These map points were compared against their known ground-truth locations, and an average error was calculated across all points.<\/li>\n<li>We validated that this average error was within our requirement (M.P.2, M.P.3)<\/li>\n<\/ul>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"size-medium wp-image-483 alignleft\" src=\"http:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/wp-content\/uploads\/sites\/48\/2020\/05\/Screen-Shot-2020-05-08-at-9.32.33-AM-234x300.png\" alt=\"\" width=\"234\" height=\"300\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/wp-content\/uploads\/sites\/48\/2020\/05\/Screen-Shot-2020-05-08-at-9.32.33-AM-234x300.png 234w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/wp-content\/uploads\/sites\/48\/2020\/05\/Screen-Shot-2020-05-08-at-9.32.33-AM-230x294.png 230w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/wp-content\/uploads\/sites\/48\/2020\/05\/Screen-Shot-2020-05-08-at-9.32.33-AM-350x448.png 350w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/wp-content\/uploads\/sites\/48\/2020\/05\/Screen-Shot-2020-05-08-at-9.32.33-AM.png 414w\" sizes=\"auto, (max-width: 234px) 100vw, 234px\" \/><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-484 alignleft\" src=\"http:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/wp-content\/uploads\/sites\/48\/2020\/05\/Screen-Shot-2020-05-08-at-9.33.48-AM-300x228.png\" alt=\"\" width=\"393\" height=\"299\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/wp-content\/uploads\/sites\/48\/2020\/05\/Screen-Shot-2020-05-08-at-9.33.48-AM-300x228.png 300w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/wp-content\/uploads\/sites\/48\/2020\/05\/Screen-Shot-2020-05-08-at-9.33.48-AM-230x175.png 230w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/wp-content\/uploads\/sites\/48\/2020\/05\/Screen-Shot-2020-05-08-at-9.33.48-AM-350x266.png 350w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/wp-content\/uploads\/sites\/48\/2020\/05\/Screen-Shot-2020-05-08-at-9.33.48-AM-480x365.png 480w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/wp-content\/uploads\/sites\/48\/2020\/05\/Screen-Shot-2020-05-08-at-9.33.48-AM.png 650w\" sizes=\"auto, (max-width: 393px) 100vw, 393px\" \/><\/p>\n<p><strong>Figure 1<\/strong>. Mapping test setup (left) and calculated euclidean error across each unit box point (right)<\/p>\n<h2><strong>SVD Test 2 (Localization Test)<\/strong><\/h2>\n<ul>\n<li>The Husky was tele-operated around the simulated parking lot<\/li>\n<li>Every 0.3 meters travelled, our script output the Husky&#8217;s calculated pose estimate and compared it against the ground truth pose<\/li>\n<li>We validated that the average pose error in both translation and rotation was within our requirement (M.P.4)<\/li>\n<\/ul>\n<h2><strong>SVD Test 3 (Lane Detection Test)<\/strong><\/h2>\n<ul>\n<li>The Husky was tele-operated around the simulated parking lot<\/li>\n<li>The calculated lane segmentation was compared against the known ground-truth lane region, and the mean intersection over union (IOU) was reported<\/li>\n<li>We validated that the mean IOU across all frames was within our requirement (M.P.6)<\/li>\n<\/ul>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-491\" src=\"http:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/wp-content\/uploads\/sites\/48\/2020\/05\/Screen-Shot-2020-05-08-at-9.41.04-AM-300x113.png\" alt=\"\" width=\"645\" height=\"243\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/wp-content\/uploads\/sites\/48\/2020\/05\/Screen-Shot-2020-05-08-at-9.41.04-AM-300x113.png 300w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/wp-content\/uploads\/sites\/48\/2020\/05\/Screen-Shot-2020-05-08-at-9.41.04-AM-768x289.png 768w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/wp-content\/uploads\/sites\/48\/2020\/05\/Screen-Shot-2020-05-08-at-9.41.04-AM-1024x386.png 1024w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/wp-content\/uploads\/sites\/48\/2020\/05\/Screen-Shot-2020-05-08-at-9.41.04-AM-830x313.png 830w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/wp-content\/uploads\/sites\/48\/2020\/05\/Screen-Shot-2020-05-08-at-9.41.04-AM-230x87.png 230w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/wp-content\/uploads\/sites\/48\/2020\/05\/Screen-Shot-2020-05-08-at-9.41.04-AM-350x132.png 350w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/wp-content\/uploads\/sites\/48\/2020\/05\/Screen-Shot-2020-05-08-at-9.41.04-AM-480x181.png 480w\" sizes=\"auto, (max-width: 645px) 100vw, 645px\" \/><\/p>\n<p><strong>Figure 2<\/strong>. Image seen from the simulated camera (left) and detected lane (right image). The IOU for this frame is shown in the top righthand corner.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Fall Validation Demonstration Results We performed two sets of full system tests for our Fall Validation Demos; one in simulation, and one [&hellip;]<\/p>\n","protected":false},"author":218,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"page-templates\/page_fullwidth.php","meta":{"footnotes":""},"class_list":["post-64","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/wp-json\/wp\/v2\/pages\/64","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/wp-json\/wp\/v2\/users\/218"}],"replies":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/wp-json\/wp\/v2\/comments?post=64"}],"version-history":[{"count":21,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/wp-json\/wp\/v2\/pages\/64\/revisions"}],"predecessor-version":[{"id":759,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/wp-json\/wp\/v2\/pages\/64\/revisions\/759"}],"wp:attachment":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teami\/wp-json\/wp\/v2\/media?parent=64"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}