{"id":32,"date":"2025-11-23T23:41:17","date_gmt":"2025-11-23T23:41:17","guid":{"rendered":"https:\/\/mrsdprojects.ri.cmu.edu\/2026teamf\/?page_id=32"},"modified":"2026-05-02T03:40:23","modified_gmt":"2026-05-02T03:40:23","slug":"system-performance","status":"publish","type":"page","link":"https:\/\/mrsdprojects.ri.cmu.edu\/2026teamf\/system-performance\/","title":{"rendered":"System Performance"},"content":{"rendered":"\n

This page summarizes the spring-semester performance results for FireSense, evaluated against the requirements targeted for the Spring Validation Demonstration (SVD). Tests were conducted at the Allegheny County Fire Academy (ACFA)<\/strong> concrete building and at the Robolounge in Newell-Simon Hall (NSH)<\/strong> for the SVD encore.<\/p>\n\n\n\n

\n\n\n\n

Spring Targeted Requirements<\/h4>\n\n\n\n

The four core capabilities prioritized this semester were: mapping the building, sensing physical barriers, localizing the payloads, and producing an accurate global map<\/strong>, along with the deployability and simple-power-supply non-functional requirements.<\/p>\n\n\n\n

Requirement<\/th>Target<\/th>Result<\/th><\/tr><\/thead>
Map the building<\/td>5 m Chamfer Distance, 0.2 Hz update<\/td>\u2705 Met<\/td><\/tr>
Sense physical barriers<\/td>0.8\u20139 m range<\/td>\u2705 Met<\/td><\/tr>
Localize payloads<\/td>1 m ATE, 45\u00b0 AOE<\/td>\u2705 Met (0.18 m ATE)<\/td><\/tr>
Create global map<\/td>1 m ATE, 45\u00b0 AOE<\/td>\u2705 Met<\/td><\/tr>
Deployable system<\/td>Field-ready, untethered<\/td>\u2705 Met (915 MHz HaLow)<\/td><\/tr>
Simple power supply<\/td>Standard battery<\/td>\u2705 Met (Makita 18V)<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\n\n\n\n

SLAM Frontend Performance<\/h4>\n\n\n\n

The MAC-VIO frontend was evaluated against an RGB-based MAC-VO baseline. Key results:<\/p>\n\n\n\n

    \n
  • Relative Translation Error (RTE):<\/strong> 0.0066 m \u2014 strong translational consistency<\/li>\n\n\n\n
  • Relative Orientation Error (ROE):<\/strong> 0.2302 rad \u2014 higher rotational drift due to limited and noisy thermal features<\/li>\n\n\n\n
  • Runtime:<\/strong> ~4 Hz on Jetson AGX Orin with TensorRT acceleration<\/li>\n<\/ul>\n\n\n\n

    The frontend provides reliable short-term tracking, while accumulated drift is corrected by the backend’s loop closure for global consistency.<\/p>\n\n\n\n

    \n\n\n\n

    SLAM Backend & Trajectory Accuracy<\/h4>\n\n\n\n

    After loop closure, the optimized pose graph dramatically reduces drift compared to raw odometry. The table below summarizes alignment RMSE (Umeyama-aligned, evaluated with evo<\/strong>) for both demonstration runs:<\/p>\n\n\n\n

    Run<\/th>ATE (m)<\/th>Orientation RMSE (\u00b0)<\/th><\/tr><\/thead>
    SVD \u2014 ACFA<\/strong> (concrete building)<\/td>0.18<\/strong><\/td>15<\/td><\/tr>
    SVD Encore \u2014 NSH<\/strong> (Robolounge)<\/td>0.21<\/td>18<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

    Both results comfortably meet the 1 m ATE<\/strong> and 45\u00b0 AOE<\/strong> targets.<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    [Sequential odometry\/VIO (red) vs. optimized pose graph after loop closures (yellow) \u2014 shows how loop closure pulls the drifted trajectory back into a globally consistent map<\/em>]<\/p>\n\n\n\n

    \n\n\n\n

    Map Quality Comparison<\/h4>\n\n\n\n

    The FireSense thermal SLAM map was compared against a baseline reconstruction generated by NVIDIA cuVSLAM<\/strong> using the ZED-X RGB stereo camera. Chamfer-distance residuals between the aligned point clouds are color-coded to show spatial agreement.<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    [Map comparison: cuVSLAM RGB dense point cloud vs. our stereo-thermal map from MAC-VIO\/SLAM, with Chamfer residuals false-colored after spatial alignment<\/em>]<\/p>\n\n\n\n

    The thermal-based reconstruction successfully captures the structural layout of the test environment, demonstrating that thermal SLAM can produce maps comparable to RGB-based methods \u2014 a critical capability for smoke-filled environments where RGB cameras fail.<\/p>\n\n\n\n

    \n\n\n\n

    Communication Performance<\/h4>\n\n\n\n

    Migrating the middleware from FastDDS multicast<\/strong> to CycloneDDS + Zenoh bridge unicast<\/strong> produced major bandwidth improvements, eliminating the network jamming caused by ROS topic discovery traffic during multi-payload operation:<\/p>\n\n\n\n

    Traffic Metric<\/th>Old (FastDDS)<\/th>New (Zenoh Bridge)<\/th>Reduction<\/th><\/tr><\/thead>
    Incoming AVG<\/td>0.49 MB\/s<\/td>0.09 MB\/s<\/td>81%<\/strong><\/td><\/tr>
    Incoming MAX<\/td>2.09 MB\/s<\/td>0.16 MB\/s<\/td>92%<\/strong><\/td><\/tr>
    Outgoing AVG<\/td>0.03 MB\/s<\/td>0.00 MB\/s<\/td>~100%<\/strong><\/td><\/tr>
    Outgoing MAX<\/td>4.30 MB\/s<\/td>0.01 MB\/s<\/td>99.7%<\/strong><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

    These improvements directly support multi-payload C-SLAM scalability and prepared the system for two-payload operation in the fall.<\/p>\n\n\n\n

    \n\n\n\n

    Summary<\/h4>\n\n\n\n

    By the end of the spring semester, FireSense successfully demonstrated:<\/p>\n\n\n\n

      \n
    • A working end-to-end thermal SLAM pipeline from sensing through global optimization<\/li>\n\n\n\n
    • Trajectory accuracy well within targeted specifications (0.18 m ATE vs. 1 m target)<\/li>\n\n\n\n
    • Reliable wireless operation through walls via 915 MHz HaLow<\/li>\n\n\n\n
    • A modular, field-deployable wearable payload with simple power supply<\/li>\n\n\n\n
    • Substantial bandwidth headroom for scaling to two-payload collaborative SLAM in the fall<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"

      This page summarizes the spring-semester performance results for FireSense, evaluated against the requirements targeted for the Spring Validation Demonstration (SVD). Tests were conducted at the Allegheny County Fire Academy (ACFA) concrete building and at the Robolounge in Newell-Simon Hall (NSH) for the SVD encore. Spring Targeted Requirements The four core capabilities prioritized this semester were: […]<\/p>\n","protected":false},"author":456,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-32","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2026teamf\/wp-json\/wp\/v2\/pages\/32","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2026teamf\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2026teamf\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2026teamf\/wp-json\/wp\/v2\/users\/456"}],"replies":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2026teamf\/wp-json\/wp\/v2\/comments?post=32"}],"version-history":[{"count":4,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2026teamf\/wp-json\/wp\/v2\/pages\/32\/revisions"}],"predecessor-version":[{"id":270,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2026teamf\/wp-json\/wp\/v2\/pages\/32\/revisions\/270"}],"wp:attachment":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2026teamf\/wp-json\/wp\/v2\/media?parent=32"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}