{"id":142,"date":"2019-01-23T03:07:52","date_gmt":"2019-01-23T03:07:52","guid":{"rendered":"http:\/\/mrsdprojects.ri.cmu.edu\/2018teami\/?page_id=142"},"modified":"2019-12-13T22:22:37","modified_gmt":"2019-12-13T22:22:37","slug":"performance","status":"publish","type":"page","link":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teami\/performance\/","title":{"rendered":"Performance"},"content":{"rendered":"

Fall Validation Performance Evaluation<\/h2>\n

Test 1: Realistic Simulator <\/strong><\/p>\n\n\n\n\n\t\n\n\t\n\t\n\t
ID <\/strong><\/th> Requirement <\/strong><\/th> FVD Performance <\/strong><\/th>\n<\/tr>\n<\/thead>\n
M.P.5<\/td>Rover is able to avoid obstacles of height at least 0.1m 80% of the time and maintains a safe distance of 0.5 m from the obstacle <\/td>Rover is successfully able to avoid obstacles of height at least .1m 100% of the times. The closest the rover comes to an obstacle is 0.7m.
\n<\/td>\n<\/tr>\n
M.P.6<\/td>Odometry error every 1m distance travelled should have a mean of at least 0.005m with a standard deviation of at least 0.002m <\/td>Odometry error every 1m distance travelled had a mean of at least 0.005m 100% of the time. The standard deviation of the same was also always within the desired range.<\/td>\n<\/tr>\n
N.F.R 1<\/td>Have realistic Simulation: Spatiotemporal information of the sun changes according to real Sun trajectory as seen from the specified latitude and longitude of the moon.<\/td>Successfully demonstrated realistic illumination modelling for four different latitude-longitude pairs. <\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n
\"\"
Figure: The graph validates that the rover maintains a safe distance of 0.5m from the rocks 100% of the time
<\/strong><\/figcaption><\/figure>\n

Test 2: Full Mission Validation <\/strong><\/p>\n\n\n\n\n\t\n\n\t\n\t\n\t\n\t
ID <\/strong><\/th> Requirement <\/strong><\/th> FVD Performance <\/strong><\/th>\n<\/tr>\n<\/thead>\n
M.P.1<\/td>Generate global waypoints around the pit within 15m from the edge of the pit for at least 80% of the waypoints using a global map resolution of 5m per pixel<\/td>The global planner successfully generated 100% of the around-pit waypoints within 15m from the edge of the pit. <\/td>\n<\/tr>\n
M.P.2<\/td>Generate vantage points according to the changing illumination and plan to reach the vantage waypoints at least 90% of the time. <\/td>The global planning for illumination based planning was successfully validated for four different pits (different latitude and longitude coordinate pairs). Accurate illumination-planning and total coverage of the pit was demonstrated. <\/td>\n<\/tr>\n
M.P.3<\/td>Locally plan to face the illuminated wall of the pit at least 70% of the time. <\/td>This requirement was validated by observing the snapped images. The rover always faced the illuminated side of the pit walls. <\/td>\n<\/tr>\n
M.P.4<\/td>The rover reaches at least 1m away from the pit edge to image the pit interior at least 60% of the time.<\/td>The rover was successfully able to navigate up-close to the pit edge within 1m 100% of the time.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n

The full mission video can be accessed at https:\/\/youtu.be\/u0G5B1FtIYM<\/a><\/p>\n

\"\"
Figure: This figure validates the local planner as it generates waypoints within 1m of the pit edge
<\/strong><\/figcaption><\/figure>\n
\"\"
Figure: The result graph indicating how far each of the generated global waypoint is from the pit edge. The red line represents the threshold of 15m. It can be observed that all waypoints are generated within that threshold
<\/strong><\/figcaption><\/figure>\n
\"\"
Figure: This figure tests the illumination-based planner of the system. The rover should be at one of the vantage points (in red) to capture the illuminated portion of the pit (in white)
<\/strong><\/figcaption><\/figure>\n
\"\"
Figure: This diagram presents the pictures captured at each vantage point, in the mission. The lit portion of the pit is clearly visible in each of the snapshots taken
<\/strong><\/figcaption><\/figure>\n

Spring Validation Performance Evaluation<\/h2>\n

\u00a0<\/strong><\/p>\n

Test 1: Characterization of Detection<\/strong><\/p>\n

This test validates the detection pipeline of the system. 30 pictures of the rover were captured using the catadioptric camera. The detection algorithm was then executed on these images. The algorithm returned a bounding box on the rover and also gave as an output the pixel value of the rover centroid. This was then compared to the ground truth. The ground truth value was found out by manually clicking on the centroid of the rover in the same image. If these two centroids were within a margin of 40 pixels, the detection was marked correct. This threshold was configured taking into account the bearing error we may obtain if the detection is within 40 pixels of the true value for a 2000 X 2000 pixels image. We had stated precision of 80% in our requirements and we were able to achieve a precision of 92%.<\/p>\n

\"\"<\/p>\n

Figure: (a) Ground truth centroid in red (b) Centroid obtained via detection algorithm in blue<\/p>\n

Test 2:<\/strong> Characterization of Range, Bearing and Communication<\/strong><\/p>\n

This test was divided into three sub-tasks:<\/p>\n

    \n
  1. The first corresponds to the validation of the range and the bearing measurement. The range measured by the UWB Decawave sensor gave a mean error of 14 cm against the stated requirement of 30cm. This was validated by driving the rover and then measuring the ground truth using a 12m measuring tape. The mean bearing error was stated at 4\u00b0<\/strong>. To validate the bearing measurement the x and y values given by the RTK GPS were taken and then the ground truth bearing was computed by applying trigonometry on the same. The following requirement was not validated completely due to unreliable and fluctuating estimates.<\/li>\n
  2. The next requirement corresponds to the successful capture of sensor data after 15 seconds. The stated requirement was 70% of the time. This requirement was validated by using the real-time visualization dashboard which depicted when the sensor measurements were recorded. We achieved a successful sensor capture accuracy of 100%.<\/li>\n
  3. The third task pertains to the validation of the communication range. We achieved a communication range of 35.3 m between the two rovers against the stated 20m range. This was validated by pinging one rover from the other. If the second rover was able to receive the ping this implies the communication is consistent.<\/li>\n<\/ol>\n

    \"\"<\/p>\n

    Test 3:<\/strong> Characterization of Range, Bearing and Communication<\/strong><\/p>\n

    This test was divided into three sub-tasks<\/p>\n

      \n
    1. The first corresponds to the validation of the RTK GPS. This was validated by using measuring tape as the ground truth. We promised the values from the RTK GPS to be within \u00b1 6cm of the true value. We were able to validate this deviation to a tighter bound of \u00b1 2.5cm<\/li>\n
    2. The next requirement corresponds to the successful completion of co-localization. We stated that we would be able to achieve an updated path of the rovers within an error of 5% of the distance traveled.<\/li>\n
    3. The third task pertains to the validation of localization using odometry\/EKF. This was expected to be within 20% of the ground truth value (measured from the RTK) for every 100m traveled. This requirement was not validated.<\/li>\n<\/ol>\n

      \"\"<\/p>\n","protected":false},"excerpt":{"rendered":"

      Fall Validation Performance Evaluation Test 1: Realistic Simulator Test 2: Full Mission Validation The full mission video can be accessed at https:\/\/youtu.be\/u0G5B1FtIYM […]<\/p>\n","protected":false},"author":175,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-142","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teami\/wp-json\/wp\/v2\/pages\/142","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teami\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teami\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teami\/wp-json\/wp\/v2\/users\/175"}],"replies":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teami\/wp-json\/wp\/v2\/comments?post=142"}],"version-history":[{"count":17,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teami\/wp-json\/wp\/v2\/pages\/142\/revisions"}],"predecessor-version":[{"id":541,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teami\/wp-json\/wp\/v2\/pages\/142\/revisions\/541"}],"wp:attachment":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teami\/wp-json\/wp\/v2\/media?parent=142"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}