{"id":309,"date":"2025-05-02T21:24:58","date_gmt":"2025-05-02T21:24:58","guid":{"rendered":"https:\/\/mrsdprojects.ri.cmu.edu\/2025teame\/?page_id=309"},"modified":"2025-05-03T00:40:56","modified_gmt":"2025-05-03T00:40:56","slug":"component-testing","status":"publish","type":"page","link":"https:\/\/mrsdprojects.ri.cmu.edu\/2025teame\/documents\/component-testing\/","title":{"rendered":"Component Testing"},"content":{"rendered":"\n

Perception Subsystem Testing<\/h2>\n\n\n\n
Pepper<\/strong><\/td>Distance Error (cm)<\/strong><\/td><\/tr>
Pepper 1<\/td>2.3<\/td><\/tr>
Pepper 2<\/td>2.9<\/td><\/tr>
Pepper 3<\/td>2.98<\/td><\/tr>
Pepper 4<\/td>1.7<\/td><\/tr><\/tbody><\/table>
Quantitative Evaluation of Fine Pose Estimation<\/strong><\/figcaption><\/figure>\n\n\n\n

<\/p>\n\n\n\n

Pepper Segmentation Results with YOLOv8 instance segmentation model<\/p>\n\n\n\n

<\/p>\n\n\n\n

Peduncle Segmentation Results with YOLOv8 instance segmentation model<\/p>\n\n\n\n

Metric<\/strong><\/td>Pepper<\/strong><\/td>Peduncle<\/strong><\/td><\/tr>
mAP@50<\/td>92.5%<\/td>92.6%<\/td><\/tr>
Precision<\/td>94.1%<\/td>96.26%<\/td><\/tr>
Recall<\/td>86.0%<\/td>85.75%<\/td><\/tr><\/tbody><\/table>
Quantitative Evaluation of Segmentation Models<\/figcaption><\/figure>\n\n\n\n

Planning Subsystem Testing<\/h2>\n\n\n\n

Dual Arm Simulation Testing<\/h3>\n\n\n\n
No.<\/strong><\/td>Result<\/strong><\/td>Simulated Pose <\/strong>(xyz, roll pitch yaw)<\/strong><\/td>Notes<\/strong><\/td><\/tr>
1.<\/strong><\/td>\u2705<\/strong><\/td>[0.28, -0.46, 0.53, 0.26, -0.26, 0.0] <\/strong><\/td>Success<\/strong><\/td><\/tr>
2<\/strong><\/td>\u2705<\/strong><\/td>[0.45, -0.63, 0.48, 0.38, 0.43, 0.0] <\/strong><\/td>Success<\/strong><\/td><\/tr>
3<\/strong><\/td>\u26a0\ufe0f<\/strong><\/td>[0.3, -0.5, 0.57, -0.46, 0.37, 0.0] <\/strong><\/td>Success, but cutter had risky pathing<\/strong><\/td><\/tr>
4<\/strong><\/td>\u2705<\/strong><\/td>[0.34, -0.56, 0.57, 0.13, 0.18, 0.0] <\/strong><\/td>Success<\/strong><\/td><\/tr>
5<\/strong><\/td>\u2705<\/strong><\/td>[0.31, -0.46, 0.46, -0.01, 0.46, 0.0] <\/strong><\/td>Success<\/strong><\/td><\/tr>
6<\/strong><\/td>\u2705<\/strong><\/td>[0.27, -0.46, 0.49, -0.04, -0.3, 0.0] <\/strong><\/td>Success<\/strong><\/td><\/tr>
7<\/strong><\/td>\u2705<\/strong><\/td>[0.27, -0.63, 0.49, -0.33, 0.27, 0.0] <\/strong><\/td>Success<\/strong><\/td><\/tr>
8<\/strong><\/td>\u2705<\/strong><\/td>[0.25, -0.54, 0.45, -0.27, -0.27, 0.0] <\/strong><\/td>Success<\/strong><\/td><\/tr>
9<\/strong><\/td>\u2705<\/strong><\/td>[0.26, -0.59, 0.45, 0.02, -0.49, 0.0] <\/strong><\/td>Success<\/strong><\/td><\/tr>
10<\/strong><\/td>\u2705<\/strong><\/td>[0.4, -0.56, 0.45, -0.03, 0.48, 0.0] <\/strong><\/td>Success<\/strong><\/td><\/tr><\/tbody><\/table>
Results of Validation Runs in Simulation<\/strong><\/figcaption><\/figure>\n\n\n\n

Overall SVD Testing<\/h2>\n\n\n\n
Sl No.<\/strong><\/td>Outcome<\/strong><\/td>Notes<\/strong><\/td><\/tr>
1.<\/strong><\/td>\u2705<\/strong><\/td> <\/strong><\/td><\/tr>
2<\/strong><\/td>\u274c<\/strong><\/td>Planning – Failed to find valid grasp plan<\/strong><\/td><\/tr>
3<\/strong><\/td>\u2705<\/strong><\/td> <\/strong><\/td><\/tr>
4<\/strong><\/td>\u2705<\/strong><\/td> <\/strong><\/td><\/tr>
5<\/strong><\/td>\u274c<\/strong><\/td>Perception – Fine Pose estimate incorrect<\/strong><\/td><\/tr>
6<\/strong><\/td>\u2705<\/strong><\/td> <\/strong><\/td><\/tr>
7<\/strong><\/td>\u2705<\/strong><\/td> <\/strong><\/td><\/tr>
8<\/strong><\/td>\u274c<\/strong><\/td>Planning – Incorrect Grasp Planning<\/strong><\/td><\/tr>
9<\/strong><\/td>\u274c<\/strong><\/td>Misc – Grasp Failure due to plastic leaves<\/strong><\/td><\/tr>
10<\/strong><\/td>\u274c<\/strong><\/td>Perception – False Positive Segmentation<\/strong><\/td><\/tr><\/tbody><\/table>
Results of Validation Runs with Hardware Setup<\/strong><\/figcaption><\/figure>\n\n\n\n

<\/p>\n\n\n\n

Functional Requirement<\/strong><\/td>ID<\/strong><\/td>Performance Requirement<\/strong><\/td>Desired<\/strong><\/td>Actual<\/strong><\/td><\/tr>
Identify, Localize and Prioritize Green Peppers<\/td>PR.01<\/td>Detect fully visible Peppers > 70% of the time<\/td>70 %<\/td>86 %<\/td><\/tr>
PR.02<\/td>Produce estimate pose of green pepper and peduncle within 3 cm of ground truth depth and within 2 cm of other coordinates and upto 30 degrees in each rotation axis<\/td>3 cm<\/td>2.4 cm<\/td><\/tr>
Harvest Green Pepper<\/td>PR.04<\/td>Reach target green peppers > 70% of the time<\/td>70 %<\/td>100 % (Sim)80 % (Real)<\/td><\/tr>
Minimize Green Pepper Damage<\/td>PR.08<\/td>Avoids deformation and damage to 90% of picked green peppers<\/td>90 %<\/td>95 %<\/td><\/tr>
Overall Success<\/td>PR.09<\/td>Avoid visible damage to harvested green pepper > 90% of the time<\/td>90 %<\/td>100 %<\/td><\/tr>
PR.10<\/td>Harvest a fully visible green pepper in testbed autonomously within 100 seconds<\/td>100 s<\/td>68 s (1-Arm)73 s (2-Arm)<\/td><\/tr><\/tbody><\/table>
Quantitative Metrics of SVD Results<\/strong><\/figcaption><\/figure>\n","protected":false},"excerpt":{"rendered":"

Perception Subsystem Testing Pepper Distance Error (cm) Pepper 1 2.3 Pepper 2 2.9 Pepper 3 2.98 Pepper 4 1.7 Quantitative Evaluation of Fine Pose Estimation Pepper Segmentation Results with YOLOv8 instance segmentation model Peduncle Segmentation Results with YOLOv8 instance segmentation […]<\/p>\n","protected":false},"author":406,"featured_media":0,"parent":25,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-309","page","type-page","status-publish","hentry","clearfix"],"_links":{"self":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2025teame\/wp-json\/wp\/v2\/pages\/309","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2025teame\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2025teame\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2025teame\/wp-json\/wp\/v2\/users\/406"}],"replies":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2025teame\/wp-json\/wp\/v2\/comments?post=309"}],"version-history":[{"count":3,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2025teame\/wp-json\/wp\/v2\/pages\/309\/revisions"}],"predecessor-version":[{"id":351,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2025teame\/wp-json\/wp\/v2\/pages\/309\/revisions\/351"}],"up":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2025teame\/wp-json\/wp\/v2\/pages\/25"}],"wp:attachment":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2025teame\/wp-json\/wp\/v2\/media?parent=309"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}