{"id":344,"date":"2020-05-05T17:27:41","date_gmt":"2020-05-05T17:27:41","guid":{"rendered":"http:\/\/mrsdprojects.ri.cmu.edu\/2020teame\/?page_id=344"},"modified":"2020-12-18T16:21:14","modified_gmt":"2020-12-18T16:21:14","slug":"testing-and-validation","status":"publish","type":"page","link":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teame\/system-performance\/testing-and-validation\/","title":{"rendered":"Testing and Validation"},"content":{"rendered":"

Spring Testing and Validation<\/h1>\n

Perception Subsystem Testing (SVD)<\/h2>\n

There were two evaluation criteria we tested our system against. The first is to localize objects within 5 meters with less than 20 cm error (M.P.6). We placed the object at various locations on a 5×4 meter grid and recorded the localization output by the sensor pod. <\/span>The localization error plot is shown below. we can see that for all locations, the maximum error is less than 20 cm. <\/span><\/p>\n

\"\"<\/p>\n

The second criteria is to detect objects with 70% accuracy (M.P.7). Besides visually inspecting the demonstration video, the mean average precision (mean average precision) plot validates this criterion as shown in Fig. 12. <\/span><\/p>\n

\"\"<\/p>\n

Planning Subsystem Testing (SVD)<\/h2>\n

Using the SITL Kinematic simulator we tested the obstacle conditions in which the drone successfully avoids obstacles. We tested the system in three different scenarios: single stationary obstacle, multiple stationary obstacles, and single moving obstacle. Our testing concluded the potential field planner is successful given the following criteria:<\/p>\n

    \n
  1. Stationary Obstacle:<\/strong>\n
      \n
    1. Gaussian sensor noise is less than 0.5 m (applied to the X, Y, and Z directions independently)<\/li>\n
    2. The detection system detects obstacles in at least 50% of motion steps.<\/li>\n<\/ol>\n<\/li>\n
    3. Multiple Obstacles:<\/strong>\n
        \n
      1. The spacing between obstacles through which the drone flies must be larger than 2x the drone’s diameter.<\/li>\n<\/ol>\n<\/li>\n
      2. Moving Obstacle:<\/strong>\n
          \n
        1. The obstacles must be moving as fast, or slower, than the drone.<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n

          The following video shows the moving and multiple object avoidance scenarios:<\/p>\n\n\n

          \nhttps:\/\/www.youtube.com\/watch?v=07k4HSxnYy0\n<\/div><\/figure>\n\n\n

          Fall Testing and Validation<\/h1>\n

          Hardware in the Loop (HITL) Testing<\/h2>\n

          The simulations in the spring used a simple kinematic controller. Using the DJI Assistant software, we simulated the motion planner and proper interfacing with the low level DJI motor controller. The motion simulation was performed using a DJI dynamic simulator. Screenshots for the results of these tests are shown below.<\/p>\n\n\n

          \n
          \n
          \"\"<\/figure>\n<\/div>\n\n\n\n
          \n
          \"\"<\/figure>\n<\/div>\n<\/div>\n\n\n\n

          Cart Testing<\/strong><\/p>\n\n\n\n

          To test the integration of both perception and planning subsystems on the final flight platform, the team conducted a series of cart tests. In these tests, the drone sat on a cart with a monitor to visualize both the live perception data and the recommended path. An operator pushed the cart along the path to simulate flight motion. Performing these tests without the overhead of flying the drone accelerated the testing process to resolve any integration issues. This picture shows the setup for this environment.<\/p>\n\n\n\n

          \"\"<\/a><\/figure><\/div>\n\n\n\n

          Finally, these tests were repeated by physically flying the drone according to the FVD specifications. Both the cart tests and the flight tests explored a wide range of testing parameters: single object, multiple objects, moving object, multiple classes, various maximum drone speeds, etc. By scaling the testing up in this manner, the team could validate each subsystem in a simplified setting until reaching the final demonstration environment.<\/p>\n","protected":false},"excerpt":{"rendered":"

          Spring Testing and Validation Perception Subsystem Testing (SVD) There were two evaluation criteria we tested our system against. The first is to localize objects within 5 meters with less than 20 cm error (M.P.6). We placed the object at various locations on a 5×4 meter grid and recorded the localization output by the sensor pod. The
          + Read More<\/a><\/p>\n","protected":false},"author":201,"featured_media":0,"parent":58,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-344","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teame\/wp-json\/wp\/v2\/pages\/344","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teame\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teame\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teame\/wp-json\/wp\/v2\/users\/201"}],"replies":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teame\/wp-json\/wp\/v2\/comments?post=344"}],"version-history":[{"count":8,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teame\/wp-json\/wp\/v2\/pages\/344\/revisions"}],"predecessor-version":[{"id":556,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teame\/wp-json\/wp\/v2\/pages\/344\/revisions\/556"}],"up":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teame\/wp-json\/wp\/v2\/pages\/58"}],"wp:attachment":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teame\/wp-json\/wp\/v2\/media?parent=344"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}