{"id":627,"date":"2020-05-07T17:55:24","date_gmt":"2020-05-07T21:55:24","guid":{"rendered":"http:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/?page_id=627"},"modified":"2020-05-07T23:26:09","modified_gmt":"2020-05-08T03:26:09","slug":"spring-validation-demonstration","status":"publish","type":"page","link":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/performance\/spring-validation-demonstration\/","title":{"rendered":"Spring Validation Demonstration"},"content":{"rendered":"<h3>Demonstration #1<\/h3>\n<p>For Demonstration #1 of the Spring Validation Demonstration (SVD), the terrain comprehension subsystem was evaluated against M.P.1 and M.P.3., which stipulate the accuracy of the segmentation of road and puddle, respectively. For road, 96.5% of frames had an IoU greater than85%. For puddle, 38.9% of frames had an IoU greater than 85%. While not a performance require-ment evaluated for SVD, the average IoU of road across all frames was 95.2%, and the averageIoU of puddle was 73.9% across all frames.\u00a0 Figure 1 shows the frame segmented into road and puddle.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-762\" src=\"http:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-content\/uploads\/sites\/41\/2020\/05\/svd_demo1.png\" alt=\"\" width=\"1280\" height=\"721\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-content\/uploads\/sites\/41\/2020\/05\/svd_demo1.png 1280w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-content\/uploads\/sites\/41\/2020\/05\/svd_demo1-300x169.png 300w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-content\/uploads\/sites\/41\/2020\/05\/svd_demo1-768x433.png 768w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-content\/uploads\/sites\/41\/2020\/05\/svd_demo1-1024x577.png 1024w\" sizes=\"auto, (max-width: 1280px) 100vw, 1280px\" \/><\/p>\n<p style=\"text-align: center\"><strong>Figure 1. <\/strong>Performance of the perception subsystem on a single frame<\/p>\n<ul>\n<li>M.P.1. &#8211; road segmentation IoU greater than 85% for at least 85% of frames <span style=\"font-weight: 400\">\u2705<\/span><\/li>\n<li>M.P.3. &#8211; puddle segmentation IoU greater than 85% for at least 85% of frames\u00a0<span style=\"font-weight: 400\">\u26d4<\/span><\/li>\n<\/ul>\n<h3>Demonstration #2<\/h3>\n<div class=\"pdf-page-container page-container ng-scope\">\n<div class=\"plv-page-view page-view\">\n<div class=\"plv-text-layer text-layer\">To display the lateral performance of curvilinear path planning and tracking control aspects of the system, a map of a U-turn was generated with a 12m straight-away leading up to a 1.5m radius, 180 degree turn, followed by another 12m straight-away shown in Figure 2. This scenario was also used to demonstrate the combined capability of the aforementioned aspects in simulation and on the robot. For Demonstration #2, the system was evaluated against M.P.5.,M.P.6., M.P.8., M.N.2., M.N.4.<\/div>\n<div>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-728\" src=\"http:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-content\/uploads\/sites\/41\/2020\/05\/demo2_pathplan-1.jpg\" alt=\"\" width=\"1596\" height=\"534\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-content\/uploads\/sites\/41\/2020\/05\/demo2_pathplan-1.jpg 1596w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-content\/uploads\/sites\/41\/2020\/05\/demo2_pathplan-1-300x100.jpg 300w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-content\/uploads\/sites\/41\/2020\/05\/demo2_pathplan-1-768x257.jpg 768w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-content\/uploads\/sites\/41\/2020\/05\/demo2_pathplan-1-1024x343.jpg 1024w\" sizes=\"auto, (max-width: 1596px) 100vw, 1596px\" \/><\/p>\n<p style=\"text-align: center\"><strong>Figure 2.<\/strong> Curvilinear path following in simulation<\/p>\n<div class=\"plv-text-layer text-layer\">\n<p>The trajectory was then passed to the controller within the simulation, and after the car traversed the map, the following performance plots in Figures 3 and 4 were generated.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-689 size-full\" src=\"http:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-content\/uploads\/sites\/41\/2020\/05\/lateral_error.jpg\" alt=\"\" width=\"1600\" height=\"1200\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-content\/uploads\/sites\/41\/2020\/05\/lateral_error.jpg 1600w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-content\/uploads\/sites\/41\/2020\/05\/lateral_error-300x225.jpg 300w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-content\/uploads\/sites\/41\/2020\/05\/lateral_error-768x576.jpg 768w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-content\/uploads\/sites\/41\/2020\/05\/lateral_error-1024x768.jpg 1024w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-content\/uploads\/sites\/41\/2020\/05\/lateral_error-285x214.jpg 285w\" sizes=\"auto, (max-width: 1600px) 100vw, 1600px\" \/><\/p>\n<\/div>\n<div style=\"text-align: center\"><strong>Figure 3. <\/strong>Plot of the lateral error vs time<\/div>\n<\/div>\n<\/div>\n<div class=\"plv-page-view page-view\">\n<div>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-690 size-full\" src=\"http:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-content\/uploads\/sites\/41\/2020\/05\/lateral_acc.jpg\" alt=\"\" width=\"1600\" height=\"1200\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-content\/uploads\/sites\/41\/2020\/05\/lateral_acc.jpg 1600w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-content\/uploads\/sites\/41\/2020\/05\/lateral_acc-300x225.jpg 300w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-content\/uploads\/sites\/41\/2020\/05\/lateral_acc-768x576.jpg 768w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-content\/uploads\/sites\/41\/2020\/05\/lateral_acc-1024x768.jpg 1024w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-content\/uploads\/sites\/41\/2020\/05\/lateral_acc-285x214.jpg 285w\" sizes=\"auto, (max-width: 1600px) 100vw, 1600px\" \/><\/p>\n<div style=\"text-align: center\"><strong>Figure 4. <\/strong>Plot of lateral acceleration vs time<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"pdf-page-container page-container ng-scope\">\n<p>As shown in Figure 3, the lateral error remains below 0.15m throughout the simulation and only briefly overshoots before returning to a low tracking error after the U-turn. The lateral acceleration signal throughout the simulation stays below 0.5g for the entirety of the simulation and does not exhibit any erratic jumps as displayed in Figure 4. The lateral GRMS was also calculated to be below 0.05. This same trajectory was then passed to the onboard computer for demonstrating real-world tracking. Due to issues with sensor calibration and state estimation, the system was unable to execute the trajectory for the SVD. As such, only requirements M.P.5., M.P.8., M.N.2., and M.N.4.were fully achieved, with M.P.8. and M.N.2. achieved in simulation, and M.P.6. not achieved.<\/p>\n<ul>\n<li><span style=\"font-weight: 400\">M.P.5. &#8211; capable of curvilinear motion planning \u2705<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">M.P.6. &#8211; real-time motion controls (greater than or equal to 55 Hz) <\/span><span style=\"font-weight: 400\">\u26d4<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">M.P.8. &#8211; less than 0.75 m from waypoints at all times\u00a0<\/span>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Simulation \u2705<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">On-Robot \u26d4<\/span><\/li>\n<\/ul>\n<\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">M.N.2. &#8211; less than 1.47 grms lateral acceleration<\/span>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Simulation\u00a0\u2705<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">On-Robot <\/span><span style=\"font-weight: 400\">\u26d4<\/span><\/li>\n<\/ul>\n<\/li>\n<li><span style=\"font-weight: 400\">M.N.4. &#8211; vehicle runs without external cables\u00a0<\/span><span style=\"font-weight: 400\">\u2705<\/span><\/li>\n<\/ul>\n<h3>Demonstration #3<\/h3>\n<p>Demonstration #3 consisted of a 30m straight-away with randomly located puddles of different radii, designed to show the system\u2019s ability to perform terrain-aware planning. Figure 5 shows the map and generated trajectory, starting from the left and ending at the goal on the right.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-718\" src=\"http:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-content\/uploads\/sites\/41\/2020\/05\/demo3_path.png\" alt=\"\" width=\"1600\" height=\"163\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-content\/uploads\/sites\/41\/2020\/05\/demo3_path.png 1600w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-content\/uploads\/sites\/41\/2020\/05\/demo3_path-300x31.png 300w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-content\/uploads\/sites\/41\/2020\/05\/demo3_path-768x78.png 768w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-content\/uploads\/sites\/41\/2020\/05\/demo3_path-1024x104.png 1024w\" sizes=\"auto, (max-width: 1600px) 100vw, 1600px\" \/><\/p>\n<p style=\"text-align: center\"><strong>Figure 5. <\/strong>Terrain aware path following in simulation<\/p>\n<p>From Figure 6 it is apparent that the vehicle slows down when approaching puddles and speeds up again after driving through. This follows from the trajectory in Figure 5 where the density of red dots correlates with intended speed, such that higher density sections are slower.This validates the ability of the system to perform terrain-aware planning followed by trajectory tracking. The sensor issues that prevented physical robot trajectory tracking during Demonstration#2 also prevented this demo from occurring on-robot. As such, requirement M.P.5. was met in simulation but not proven in the real-world, and M.P.6. was not achieved. Open loop control was achieved, which validated requirement M.N.4.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-723\" src=\"http:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-content\/uploads\/sites\/41\/2020\/05\/vehicle_speed.png\" alt=\"\" width=\"708\" height=\"561\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-content\/uploads\/sites\/41\/2020\/05\/vehicle_speed.png 708w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-content\/uploads\/sites\/41\/2020\/05\/vehicle_speed-300x238.png 300w\" sizes=\"auto, (max-width: 708px) 100vw, 708px\" \/><\/p>\n<p style=\"text-align: center\"><strong>Figure 6. <\/strong>Plot of vehicle speed during the simulation of Demonstration 3<\/p>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">M.P.5. &#8211; slows down for puddles<\/span>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Simulation\u00a0\u2705<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">On-Robot <\/span><span style=\"font-weight: 400\">\u26d4<\/span><\/li>\n<\/ul>\n<\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">M.P.6. &#8211; real-time motion controls (greater than or equal to 55Hz) <\/span><span style=\"font-weight: 400\">\u26d4<\/span><\/li>\n<li><span style=\"font-weight: 400\">M.N.4. &#8211; vehicle runs without external cables\u00a0<\/span><span style=\"font-weight: 400\">\u2705<\/span><\/li>\n<\/ul>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>Demonstration #1 For Demonstration #1 of the Spring Validation Demonstration (SVD), the terrain comprehension subsystem was evaluated against M.P.1 and M.P.3., which stipulate the accuracy of the segmentation of road and puddle, respectively. For road, 96.5% of frames had an IoU greater than85%. For puddle, 38.9% of frames had an IoU greater than 85%. While <a href=\"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/performance\/spring-validation-demonstration\/\" rel=\"nofollow\"><span class=\"sr-only\">Read more about Spring Validation Demonstration<\/span>[&hellip;]<\/a><\/p>\n","protected":false},"author":187,"featured_media":0,"parent":34,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-627","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-json\/wp\/v2\/pages\/627","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-json\/wp\/v2\/users\/187"}],"replies":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-json\/wp\/v2\/comments?post=627"}],"version-history":[{"count":33,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-json\/wp\/v2\/pages\/627\/revisions"}],"predecessor-version":[{"id":773,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-json\/wp\/v2\/pages\/627\/revisions\/773"}],"up":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-json\/wp\/v2\/pages\/34"}],"wp:attachment":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamb\/wp-json\/wp\/v2\/media?parent=627"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}