{"id":14,"date":"2019-12-18T03:37:16","date_gmt":"2019-12-18T03:37:16","guid":{"rendered":"http:\/\/mrsdprojects.ri.cmu.edu\/2020teamj\/?page_id=14"},"modified":"2020-12-14T05:56:46","modified_gmt":"2020-12-14T05:56:46","slug":"system-performance","status":"publish","type":"page","link":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamj\/system-performance\/","title":{"rendered":"Performance"},"content":{"rendered":"

Fall Validation Demonstration<\/strong><\/span><\/h3>\n

The Fall Validation Demonstration was conducted in two parts, in simulation, and on hardware. The following sections detail the tests performed and results obtained for both demos.<\/span><\/p>\n

Simulation<\/span><\/p>\n

In the simulation, we showcased the chassis’ ability to plan a path to the desired pod, identify it, and dock with it. It would then drop the pod off at a predefined undocking location. The validation criteria and results for the demo were:<\/span><\/p>\n

    \n
  1. The system will continuously log the system state and health. State transitions occur only when the last action has been completed. The logs will present system state, health, and results of actions in an easily readable manner.<\/strong><\/span><\/li>\n<\/ol>\n

    The Behavioural State Machine node and the Health Monitoring System operated continuously throughout the length of the simulation. The system state was logged by the State Machine. It outputted the current time, current operation mode (pickup, drop-off, idle), current state, the action performed and its output, and the state the system will transition into as a result of the action taken. The Health Monitoring System also outputted the results of sanity and persistence checks for all nodes and sensors.<\/p>\n

    \u00a0 \u00a0 2.\u00a0 (Un)Docking will be done within 120s.<\/strong><\/p>\n

    This criterion satisfies M.P.2<\/strong>. 4 processes are subsumed within this requirement: approach navigation to either pick up or drop-off the pod, final pose verification when the approach is complete, retrace if the pose achieved is outside error bounds, and the physical act of docking and lifting up the pod or undocking and placing it on the ground. The chassis executed all the procedures well within 120s in all scenarios including those in which the chassis had to retrace its path to the start of the Payload Handling Zone (PHZ) except when approach navigation fails due to a large difference in the initial pose of the chassis from the ideal initial pose at the start of the PHZ. In order to induce this failure, we had to manually shift the chassis in the simulation from the pose it had achieved at the end of the point-to-point (P2P) navigation. Our P2P algorithm ensures that the probability of occurrence of such cases is low.<\/p>\n

    \u00a0 \u00a0 3. \u00a0The simulation should have a success rate of 85%.<\/strong><\/p>\n

    The success criterion subsumes M.P.1<\/strong> (chassis will identify pod when it is within 2m of the PHZ), M.P.3<\/strong> (chassis will achieve docking pose with an error margin of \u00b15 cm in position and \u00b15.54 degrees in orientation), and M.P.8<\/strong> (chassis will identify PHZ 90% of the time). During testing, the P2P navigation was always successful, and the chassis was always able to detect it’s proximity to the PHZ. The chassis was able to achieve a docking pose within the error bounds imposed by M.P.3 either on the first try or after retracing in 90% of the cases. The system was also able to detect, report, and diagnose system failures within 2000ms (M.P.5). It also maintained a minimum distance of 30cm at all times from objects (M.P.7) and determined their pose with an accuracy of 5cm (M.P.6). Overall, the entire process was successful 7 out of 8 times (87.5% success rate<\/strong>).<\/p>\n

    Table 1 below lists the unit tests that were performed to validate each of the mandatory performance requirements along with the success rate of the test. \u00a0These tests were performed in addition to the Fall Validation demonstration to ensure the successful operation of the entire system.<\/span><\/p>\n

    Hardware<\/span><\/p>\n

    We also validated some of the algorithms we had developed with real-world data using a makeshift mobile platform with sensors mounted on top of it. The metrics that were verified and the results are described below.<\/p>\n

      \n
    1. Predicted pod position (x,y) for relative localization is within 20 cm of the ground-truth, and the orientation (yaw) is within \u00b111.80 degrees of\u00a0<\/strong>the ground-truth. The same margins hold for the predicted docking pose (x,y, yaw) error and actual error.<\/strong><\/li>\n<\/ol>\n

      For this test, we first placed the mobile platform in front of the pod where it was able to view the AprilTags mounted on the pod. The system was able to predict the pod’s detected location to within the accuracy bounds mentioned above even when the platform was rotated and translated in different directions. The platform was then moved underneath the pod. It was able to predict the docking pose within the bounds as well.<\/span><\/p>\n

      \u00a0 \u00a0 2. \u00a0Object locations are detected with an accuracy of \u00b1 5 cm (M.P.6).<\/strong><\/p>\n

      This requirement was met. We also demonstrated the pedestrian tracking ability of our system. Our system was able to identify moving pedestrians and differentiate them from static obstacles. It was also able to reassign them the same ID when they reentered the camera\u2019s field of view.<\/p>\n

      Table 1: FVD Performance Evaluation<\/b><\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
      \n

      Tests<\/span><\/h4>\n<\/td>\n

      		\n

      Demo \/ Requirement<\/span><\/h4>\n<\/td>\n

      		\n

      Success Rate<\/span><\/h4>\n<\/td>\n<\/tr>\n

      \n

      1.<\/span><\/h3>\n

      Navigation<\/span><\/h3>\n<\/td>\n

      		\n

      Path Planned to PHZ<\/span><\/h3>\n<\/td>\n

      		\n

      10\/10<\/span><\/h3>\n<\/td>\n<\/tr>\n

      \n

      Identifying proximity of ~ 2 m to PHZ (MP1)<\/span><\/h3>\n<\/td>\n

      		\n

      10\/10<\/span><\/h3>\n<\/td>\n<\/tr>\n

      \n

      2.1 Approach<\/span><\/h3>\n<\/td>\n

      		\n

      Pod identified correctly (MP requires\u00a0 > 90%) (MP8)<\/span><\/h3>\n<\/td>\n

      		\n

      10\/10<\/span><\/h3>\n<\/td>\n<\/tr>\n

      \n

      Vehicle completes approach to docking position.<\/span><\/h3>\n<\/td>\n

      		\n

      8\/10<\/span><\/h3>\n<\/td>\n<\/tr>\n

      \n

      Max error of 5 cm in the docking position<\/span> (MP3)<\/span><\/span><\/h3>\n<\/td>\n

      		\n

      10\/10<\/span><\/h3>\n<\/td>\n<\/tr>\n

      \n

      Max error of 5.54 degrees in the docking orientation (MP3)<\/span><\/h3>\n<\/td>\n

      		\n

      10\/10<\/span><\/h3>\n<\/td>\n<\/tr>\n

      \n

      2.2 Retracing<\/span><\/h3>\n<\/td>\n

      		\n

      Identify misalignment and perform retracing (MP2)<\/span><\/h3>\n<\/td>\n

      		\n

      9\/10<\/span><\/h3>\n<\/td>\n<\/tr>\n

      \n

      Max error of 5 cm in the docking position (MP3)<\/span><\/h3>\n<\/td>\n

      		\n

      10\/10<\/span><\/h3>\n<\/td>\n<\/tr>\n

      \n

      Max error of 5.54 degrees in the docking orientation (MP3)<\/span><\/h3>\n<\/td>\n

      		\n

      10\/10<\/span><\/h3>\n<\/td>\n<\/tr>\n

      3.1 HMS<\/span><\/td>\n\n

      Failure diagnosis and log within 2000 ms (MP5)<\/span><\/h3>\n<\/td>\n

      		\n

      10\/10<\/span><\/h3>\n<\/td>\n<\/tr>\n

      \n

      The system is able to log recovery of nodes and topics<\/span><\/h3>\n<\/td>\n

      		\n

      10\/10<\/span><\/h3>\n<\/td>\n<\/tr>\n

      3.2 Object detection<\/span><\/td>\n\n

      Detect object location with an accuracy of \u00b15 cm (MP6)<\/span><\/h3>\n<\/td>\n

      		\n

      10\/10<\/span><\/h3>\n<\/td>\n<\/tr>\n

      \n

      Maintain minimum distance of 30 cm from obstacles (MP7)<\/span><\/h3>\n<\/td>\n

      		\n

      10\/10<\/span><\/h3>\n<\/td>\n<\/tr>\n

      4. Docking\/ Undocking<\/span><\/td>\n\n

      This test along with Test 2.1 are completed within 120s (MP2)<\/span><\/h3>\n<\/td>\n

      		\n

      10\/10<\/span><\/h3>\n<\/td>\n<\/tr>\n

      \n

      State change request is raised with trigger condition and logged.<\/span><\/h3>\n<\/td>\n

      		\n

      10\/10<\/span><\/h3>\n<\/td>\n<\/tr>\n

      \n

      Docking\/undocking mechanism is successfully executed and the pod is lifted up or lowered in a stable manner depending on system state.<\/span><\/h3>\n<\/td>\n

      		\n

      10\/10<\/span><\/h3>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

      Spring Validation Demonstration<\/strong><\/span><\/h3>\n

      Targeted Requirements<\/span><\/p>\n

      For the Spring, we targeted the requirements that relate to the process of navigating to the pod and then aligning with it. Detailed requirements can be found here<\/a>.<\/span><\/p>\n

      Table 1: Targetted Requirements<\/b><\/p>\n\n\n\n\n\n\n\n\n\n\n
      MP<\/b><\/td>\nDescription<\/b><\/td>\nSubsystem<\/b><\/td>\nSystem Element<\/b><\/td>\n<\/tr>\n
      1<\/span><\/td>\nPlan path < 120 km<\/span><\/td>\nNavigation<\/span><\/td>\nMission Planner<\/span><\/td>\n<\/tr>\n
      2<\/span><\/td>\nIdentify proximity to PHZ<\/span><\/td>\nNavigation, Sensing<\/span><\/td>\nLocalization<\/span><\/td>\n<\/tr>\n
      3<\/span><\/td>\nDock < 120 s<\/span><\/td>\nNavigation, Docking, Sensing<\/span><\/td>\nMotion Planning, Align, Verify<\/span><\/td>\n<\/tr>\n
      4<\/span><\/td>\nAlign within the error margin<\/span><\/td>\nNavigation, Docking, Sensing<\/span><\/td>\nLocalization, Verify, Mission Planner<\/span><\/td>\n<\/tr>\n
      6<\/span><\/td>\nDiagnose failures < 2 s<\/span><\/td>\nSafety<\/span><\/td>\nHealth Monitoring System (HMS)<\/span><\/td>\n<\/tr>\n
      7<\/span><\/td>\nDetect objects<\/span><\/td>\nSensing, Navigation<\/span><\/td>\nDetection<\/span><\/td>\n<\/tr>\n
      8<\/span><\/td>\nStop > 30 cm from obstacles<\/span><\/td>\nSensing, Navigation, Safety<\/span><\/td>\nDetection, HMS<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

      Performance<\/strong><\/p>\n

      The following table lists the tests we had planned for the Spring Validation Demonstration<\/strong>, their description, and the results that we obtained in our preliminary tests as well as in both the validation demonstrations.<\/span><\/p>\n

      The video can be found\u00a0here<\/a>.<\/p>\n

      Table 2: SVD Performance Evaluation<\/b><\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
      Tests<\/strong><\/td>\nDemo\/Requirement<\/strong><\/td>\nResult (out of 5 tries)<\/strong><\/td>\n<\/tr>\n
      \n
        \n
      1. Navigation<\/span><\/li>\n<\/ol>\n<\/td>\n
      		Path Planned to PHZ <\/span>(M.P.1)<\/b><\/td>\nSuccessful – 5\/5<\/span><\/td>\n<\/tr>\n
      Identifying proximity of ~ 2 m to PHZ <\/span>(M.P.2)<\/b><\/td>\nSuccessful – 5\/5<\/span><\/td>\n<\/tr>\n
      \n
        \n
      1. Safety<\/span><\/li>\n<\/ol>\n<\/td>\n
      		MP Failure diagnosis and log within 2000 ms <\/span>(M.P.6)<\/b><\/td>\nSuccessful – 5\/5<\/span><\/td>\n<\/tr>\n
      Detect obstacles with an accuracy of 5 cm <\/span>(M.P.7)<\/b><\/td>\nDetected – 5\/5<\/span><\/p>\n

      With 5 cm accuracy – 3\/5<\/span><\/td>\n<\/tr>\n

      Maintain a distance of 30 cm from obstacles <\/span>(M.P.8)<\/b><\/td>\nSuccessful – 5\/5<\/span><\/td>\n<\/tr>\n
      \n
        \n
      1. Alignment<\/span><\/li>\n<\/ol>\n<\/td>\n
      		Dock within 120 s <\/span>(M.P.3)<\/b><\/td>\nSuccessful – 5\/5<\/span><\/p>\n

      Without Retrace: 26.5 s<\/span><\/p>\n

      With Retrace: 65.5 s<\/span><\/td>\n<\/tr>\n

      Max error of 5 cm in docking position<\/span> (M.P.4)<\/b><\/td>\n3.2 cm error on average<\/span><\/td>\n<\/tr>\n
      Max error of 5.54 degrees in docking orientation <\/span>(M.P.4)<\/b><\/td>\n0.284 degrees average<\/span><\/td>\n<\/tr>\n
      \n
        \n
      1. Retracing<\/span><\/li>\n<\/ol>\n<\/td>\n
      		Identify misalignment and perform retracing\u00a0<\/span><\/td>\nSuccessful 4\/5<\/span><\/td>\n<\/tr>\n
      Max error of 5 cm in docking position <\/span>(M.P.4)<\/b><\/td>\n4.6 cm on average<\/span><\/td>\n<\/tr>\n
      Max error of 5.54 degrees in docking orientation <\/span>(M.P.4)<\/b><\/td>\n0.3 degrees average<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

      Detailed Tests Performed<\/strong><\/p>\n

      This section will give the detailed tests performed and which subsystems and requirements were validated.<\/p>\n

      1. Navigation<\/strong><\/p>\n

      Subsystems Validated: Simulation and Navigation Subsystem (Localization, Detection, and Point to Point Navigation)<\/p>\n

      Table 3: Navigation Subsystem Tests<\/b><\/p>\n\n\n\n\n\n\n\n\n\n\n
      No.<\/b><\/td>\nTest<\/b><\/td>\nExpected Outcome<\/b><\/td>\nMPs<\/b><\/td>\n<\/tr>\n
      1.1<\/span><\/td>\nLaunch Gazebo world simulation, l<\/span>oad world map (Rviz), spawn pod.\u00a0<\/span><\/td>\nGazebo world launched, vehicle has spawned, pod has spawned and map has been loaded.<\/span><\/td>\n<\/td>\n<\/tr>\n
      1.2<\/span><\/td>\nStart Autoware localization and planning nodes.<\/span><\/td>\nLocalization visible. PHZ and its start location based on prior visible in Rviz.<\/span><\/td>\n<\/td>\n<\/tr>\n
      1.3<\/span><\/td>\nPlan path till pod, load waypoints.<\/span><\/td>\nPlanned path visible on map.<\/span><\/td>\nM.P.1<\/span><\/td>\n<\/tr>\n
      1.4<\/span><\/td>\nExecute path<\/span><\/td>\nChassis starts moving towards goal.<\/span><\/td>\nM.P.1<\/span><\/td>\n<\/tr>\n
      1.5<\/span><\/td>\nPlace obstacles in the path of the vehicle.<\/span><\/td>\nThe vehicle stops before collision occurs.<\/span><\/td>\nM.P.7, M.P.8<\/span><\/td>\n<\/tr>\n
      1.6<\/span><\/td>\nIdentify PHZ when within 2m from start point.<\/span><\/td>\nChassis halts when PHZ is detected.<\/span><\/td>\nM.P.2<\/span><\/td>\n<\/tr>\n
      1.7<\/span><\/td>\nIdentify the pod and predict its location based on laser scan data.<\/span><\/td>\nAprilTags 1 and 2 identified. Pod location estimated. Waypoints for further motion visible.<\/span><\/td>\nM.P. 9<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

      2. Safety<\/strong><\/p>\n

      Subsystems Validated: Safety<\/p>\n

      Table 4: Safety Subsystem Tests<\/b><\/p>\n\n\n\n\n\n\n\n\n
      No.<\/b><\/td>\nTest<\/b><\/td>\nExpected Outcome<\/b><\/td>\nMPs<\/b><\/td>\n<\/tr>\n
      2.1<\/span><\/td>\nManually crash obstacle-avoidance node.<\/span><\/td>\nPersistence check detects crashed node within 2s<\/span><\/td>\nM.P.6<\/span><\/td>\n<\/tr>\n
      2.2<\/span><\/td>\nManually stop publishing to NDT-pose topic<\/span><\/td>\nPersistence check detects crashed node within\u00a02s<\/span><\/td>\nM.P.6<\/span><\/td>\n<\/tr>\n
      2.3<\/span><\/td>\nRestart NDT-pose after introducing misalignment.<\/span><\/td>\nSanity check detects wrong pose info\u00a0within 2s<\/span><\/td>\nM.P.6<\/span><\/td>\n<\/tr>\n
      2.4<\/span><\/td>\nPublish incorrect lidar data.<\/span><\/td>\nSanity check detects incorrect data.<\/span><\/td>\nM.P.6<\/span><\/td>\n<\/tr>\n
      2.5<\/span><\/td>\nTry to cause collision using tele-op.<\/span><\/td>\nObstacle detection node prevents collision from occurring via HMS<\/span><\/td>\nM.P.7, M.P.8<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

      3. Align\u00a0<\/strong><\/p>\n

      Subsystems Validated: Navigation (Approach Navigation) and Docking<\/p>\n

      Table 5: Align Navigation Tests<\/b><\/p>\n\n\n\n\n\n\n\n\n\n
      No.<\/b><\/td>\nTest<\/b><\/td>\nExpected Outcome<\/b><\/td>\nMPs<\/b><\/td>\n<\/tr>\n
      3.1<\/span><\/td>\nLaunch a different simulation for docking.<\/span><\/td>\nGazebo world launched, vehicle has spawned, pod has spawned with the Apriltags loaded.<\/span><\/td>\n<\/td>\n<\/tr>\n
      3.2<\/span><\/td>\nStart 2D LIDAR-based localization and planning nodes, load pod location prior estimate.<\/span><\/td>\nLocalization visible. PHZ and its start location based on prior visible.<\/span><\/td>\n<\/td>\n<\/tr>\n
      3.3<\/span><\/td>\nGenerate set of waypoints for approach navigation procedure.<\/span><\/td>\nWaypoints will be generated and visible in RViz visualization.<\/span><\/td>\n<\/td>\n<\/tr>\n
      3.4<\/span><\/td>\nExecute approach navigation using pure pursuit algorithm and PD velocity control.<\/span><\/td>\nThe chassis will follow the set of waypoints and align itself under the pod.<\/span><\/td>\n<\/td>\n<\/tr>\n
      3.5<\/span><\/td>\nPerform pose verification using the laser scan data.<\/span><\/td>\nThe estimated pose error and docking error will be shown on the screen.<\/span><\/td>\nM.P.4<\/span><\/td>\n<\/tr>\n
      3.6<\/span><\/td>\nPerform docking once again with misalignment<\/span><\/td>\nFinal estimated pose and docking error are displayed on screen.<\/span><\/td>\nM.P.4<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

      4. Retracing<\/strong><\/p>\n

      Subsystems Validated: Docking<\/p>\n

      Table 6: Retracing Tests<\/b><\/p>\n\n\n\n\n
      No.<\/b><\/td>\nTest<\/b><\/td>\nExpected Outcome<\/b><\/td>\nMPs<\/b><\/td>\n<\/tr>\n
      4.1<\/span><\/td>\nIntroduce misalignment wrt the pod to begin retrace and dock again<\/span><\/td>\nThe bot uses the error estimate to identify that it is misaligned and starts retracing its path to the start of the PHZ.<\/span><\/p>\n

      It then begins to align again with the pod by navigating in PHZ<\/span><\/td>\n

      		M.P3, M.P.4<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n","protected":false},"excerpt":{"rendered":"

      Fall Validation Demonstration The Fall Validation Demonstration was conducted in two parts, in simulation, and on hardware. The following sections detail the tests performed and results obtained for both demos. Simulation In the simulation, we showcased the chassis’ ability to plan a path to the desired pod, identify it, and dock with it. It would
      + Read More<\/a><\/p>\n","protected":false},"author":224,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-14","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamj\/wp-json\/wp\/v2\/pages\/14","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamj\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamj\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamj\/wp-json\/wp\/v2\/users\/224"}],"replies":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamj\/wp-json\/wp\/v2\/comments?post=14"}],"version-history":[{"count":19,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamj\/wp-json\/wp\/v2\/pages\/14\/revisions"}],"predecessor-version":[{"id":378,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamj\/wp-json\/wp\/v2\/pages\/14\/revisions\/378"}],"wp:attachment":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamj\/wp-json\/wp\/v2\/media?parent=14"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}