{"id":808,"date":"2022-05-06T21:08:03","date_gmt":"2022-05-07T01:08:03","guid":{"rendered":"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/?page_id=808"},"modified":"2022-12-11T10:35:20","modified_gmt":"2022-12-11T15:35:20","slug":"system-performance","status":"publish","type":"page","link":"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/system-performance\/","title":{"rendered":"System Performance"},"content":{"rendered":"\n<p style=\"font-size:24px\"><strong>Perception Subsystem<\/strong><\/p>\n\n\n\n<p><strong>Pose Estimation<\/strong><\/p>\n\n\n\n<p>AR markers were used as fiducial markers for pose estimation since they had better ROS support and allowed us to estimate the pose of the tag in 3D space using information from the RGB image as well as the depth image. Analysis and unit testing of pose estimation was performed by setting up a large fiducial marker on a vice at a known distance from the camera frame, and oriented at a right angle along one axis as shown in figure 1. The pose estimation ROS Node was executed for 10 image frames and the mean of the estimates was taken to verify against our performance requirements.<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/05\/ar_tag_testing-1024x815.jpg\" alt=\"\" class=\"wp-image-833\" width=\"512\" height=\"408\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/05\/ar_tag_testing-1024x815.jpg 1024w, https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/05\/ar_tag_testing-300x239.jpg 300w, https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/05\/ar_tag_testing-768x611.jpg 768w, https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/05\/ar_tag_testing-1536x1222.jpg 1536w, https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/05\/ar_tag_testing.jpg 1600w\" sizes=\"auto, (max-width: 512px) 100vw, 512px\" \/><\/figure><\/div>\n\n\n\n<p class=\"has-text-align-center\"><strong>Fig 1<\/strong>. Unit testing setup of the pose estimation framework<\/p>\n\n\n\n<p><strong>Evaluation Metric:<\/strong> According to <strong>M.P.R.1<\/strong> and <strong>M.P.R.2<\/strong>, the container pose estimation algorithm should detect position with an error of &lt; 10 mm and container orientation with an error of &lt; 5\u00b0.<br><strong>Achieved metric:<\/strong> Our system detects container pose with an error of &lt; 5 mm and container orientation with an error of &lt; 2\u00b0.<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-large is-resized\"><img decoding=\"async\" src=\"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/05\/pose_test_results-edited.png\" alt=\"\" class=\"wp-image-835\" width=\"679\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/05\/pose_test_results-edited.png 1580w, https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/05\/pose_test_results-edited-300x169.png 300w, https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/05\/pose_test_results-edited-1024x578.png 1024w, https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/05\/pose_test_results-edited-768x434.png 768w, https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/05\/pose_test_results-edited-1536x867.png 1536w\" sizes=\"(max-width: 1580px) 100vw, 1580px\" \/><\/figure><\/div>\n\n\n\n<p class=\"has-text-align-center\"><strong>Fig 2.<\/strong> Evaluation of the average position and orientation error over 10 attempts<\/p>\n\n\n\n<p><strong>Ingredient Validation<\/strong><\/p>\n\n\n\n<p>Unit testing of this ingredient validation was majorly performed by verifying the validation metrics while training the model, and verifying that the ROS Service correctly generates the expected prediction during inference while in operation.<\/p>\n\n\n\n<p><strong>Evaluation Metric<\/strong>: According to <strong>M.P.R.3<\/strong>, the system will validate ingredients stored in a fixed set of translucent containers with a precision of over <strong>85%<\/strong> and recall of over <strong>90%<\/strong>.<\/p>\n\n\n\n<p><strong>Achieved metric<\/strong>: Our system validates ingredients stored in a fixed set of translucent containers with a precision of over <strong>99%<\/strong> and recall of over <strong>99%<\/strong>.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"697\" height=\"251\" src=\"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/05\/train_results.png\" alt=\"\" class=\"wp-image-836\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/05\/train_results.png 697w, https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/05\/train_results-300x108.png 300w\" sizes=\"auto, (max-width: 697px) 100vw, 697px\" \/><\/figure>\n\n\n\n<p class=\"has-text-align-center\"><strong>Fig 3.<\/strong> Validation Precision &amp; Recall over each training epoch<\/p>\n\n\n\n<p><strong>Container Weight Estimation using Force-Torque Sensor<\/strong><\/p>\n\n\n\n<p>We used the UR5e pendant\u2019s software calibration sets to perform the initial calibration. We obtained the tool center point, orientation, pose, center of gravity after the calibration was performed. This calibration was done to prevent false protective stops.<\/p>\n\n\n\n<p><br>We obtained the joint torque along X, Y, Z axes directly from the values published to ROS topics by the UR5e driver. We isolated the torque values along X axis and used the mass-torque relationship to compute the container weight;<\/p>\n\n\n\n<p class=\"has-text-align-center\"><br>Torque = m x g x L<\/p>\n\n\n\n<p><br>Where,<br>m: mass of the container<br>g: Acceleration due to gravity<br>L: Length of the moment arm (distance from the wrist joint to center of mass of the container)<br><\/p>\n\n\n\n<p>The weight of the container obtained using this method gives us a weight estimate within +-50 grams of the actual container weight.<\/p>\n\n\n\n<p style=\"font-size:24px\"><strong>Controls Subsystem<\/strong><\/p>\n\n\n\n<p>The controller being used for dispensing was evaluated by running multiple trials requesting the system to dispense different quantities of ingredients within the typical request range of the ingredient. The evaluations were done with the containers being at different fill levels for each trial. The metric used for evaluation was the number of times that the dispensing process was out of tolerance. The evaluation results are shown in the graphs below. The green circles indicate trials that were within tolerance and the red circles indicate trials that were out of tolerance.<\/p>\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-9d6595d7 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"359\" height=\"269\" src=\"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/12\/1.png\" alt=\"\" class=\"wp-image-952\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/12\/1.png 359w, https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/12\/1-300x225.png 300w\" sizes=\"auto, (max-width: 359px) 100vw, 359px\" \/><\/figure>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"354\" height=\"266\" src=\"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/12\/2.png\" alt=\"\" class=\"wp-image-954\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/12\/2.png 354w, https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/12\/2-300x225.png 300w\" sizes=\"auto, (max-width: 354px) 100vw, 354px\" \/><\/figure>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"354\" height=\"266\" src=\"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/12\/8-1.png\" alt=\"\" class=\"wp-image-966\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/12\/8-1.png 354w, https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/12\/8-1-300x225.png 300w\" sizes=\"auto, (max-width: 354px) 100vw, 354px\" \/><\/figure>\n<\/div>\n<\/div>\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-9d6595d7 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"354\" height=\"266\" src=\"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/12\/4.png\" alt=\"\" class=\"wp-image-956\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/12\/4.png 354w, https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/12\/4-300x225.png 300w\" sizes=\"auto, (max-width: 354px) 100vw, 354px\" \/><\/figure>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"409\" height=\"308\" src=\"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/12\/5.png\" alt=\"\" class=\"wp-image-957\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/12\/5.png 409w, https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/12\/5-300x226.png 300w\" sizes=\"auto, (max-width: 409px) 100vw, 409px\" \/><\/figure>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"359\" height=\"269\" src=\"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/12\/6.png\" alt=\"\" class=\"wp-image-958\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/12\/6.png 359w, https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/12\/6-300x225.png 300w\" sizes=\"auto, (max-width: 359px) 100vw, 359px\" \/><\/figure>\n<\/div>\n<\/div>\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-9d6595d7 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"393\" height=\"270\" src=\"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/12\/7.png\" alt=\"\" class=\"wp-image-959\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/12\/7.png 393w, https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/12\/7-300x206.png 300w\" sizes=\"auto, (max-width: 393px) 100vw, 393px\" \/><\/figure>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"344\" height=\"231\" src=\"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/12\/3-1.png\" alt=\"\" class=\"wp-image-967\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/12\/3-1.png 344w, https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/12\/3-1-300x201.png 300w\" sizes=\"auto, (max-width: 344px) 100vw, 344px\" \/><\/figure>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"393\" height=\"273\" src=\"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/12\/10-1.png\" alt=\"\" class=\"wp-image-968\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/12\/10-1.png 393w, https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/12\/10-1-300x208.png 300w\" sizes=\"auto, (max-width: 393px) 100vw, 393px\" \/><\/figure>\n<\/div>\n<\/div>\n\n\n\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-9d6595d7 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"359\" height=\"269\" src=\"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/12\/9-1.png\" alt=\"\" class=\"wp-image-969\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/12\/9-1.png 359w, https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/12\/9-1-300x225.png 300w\" sizes=\"auto, (max-width: 359px) 100vw, 359px\" \/><\/figure><\/div>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"354\" height=\"266\" src=\"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/12\/11.png\" alt=\"\" class=\"wp-image-963\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/12\/11.png 354w, https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-content\/uploads\/sites\/55\/2022\/12\/11-300x225.png 300w\" sizes=\"auto, (max-width: 354px) 100vw, 354px\" \/><\/figure><\/div>\n<\/div>\n<\/div>\n\n\n\n<p style=\"font-size:24px\"><strong>Overall System<\/strong><\/p>\n\n\n\n<p><strong>Spillage<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>Evaluation Metric<\/strong>: According to M.P.R.6, the system ensure ingredient spillage to be less than <strong>5%<\/strong> by weight<\/li><li><strong>Achieved metric<\/strong>: Our system ensures ingredient spillage to be less than <strong>1%<\/strong> by weight<\/li><\/ul>\n\n\n\n<p><strong>Cycle Time<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>Evaluation Metric<\/strong>: According to M.P.R.7, the system will dispense any ingredient in under <strong>3 minutes<\/strong><\/li><li><strong>Achieved metric<\/strong>: Our system dispenses any ingredient in under <strong>2 minutes<\/strong><\/li><\/ul>\n\n\n\n<p><strong>Payload<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>Evaluation Metric<\/strong>: According to M.P.R.8, the system will handle containers weighing<strong> <\/strong>up to <strong>1.5 Kgs<\/strong><\/li><li><strong>Achieved metric<\/strong>: Our system can handle containers weighing<strong> <\/strong>up to <strong>1.5 Kgs<\/strong><\/li><\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Perception Subsystem Pose Estimation AR markers were used as fiducial markers for pose estimation since they had better ROS support and allowed us to estimate the pose of the tag in 3D space using information from the RGB image as well as the depth image. Analysis and unit testing of pose estimation was performed by<br \/><a class=\"moretag\" href=\"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/system-performance\/\">+ Read More<\/a><\/p>\n","protected":false},"author":256,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"template_parts\/page-fullwidth_template.php","meta":{"footnotes":""},"class_list":["post-808","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-json\/wp\/v2\/pages\/808","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-json\/wp\/v2\/users\/256"}],"replies":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-json\/wp\/v2\/comments?post=808"}],"version-history":[{"count":9,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-json\/wp\/v2\/pages\/808\/revisions"}],"predecessor-version":[{"id":971,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-json\/wp\/v2\/pages\/808\/revisions\/971"}],"wp:attachment":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2022teamb\/wp-json\/wp\/v2\/media?parent=808"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}