{"id":22,"date":"2025-02-08T03:03:42","date_gmt":"2025-02-08T03:03:42","guid":{"rendered":"https:\/\/mrsdprojects.ri.cmu.edu\/2025teamg\/?page_id=22"},"modified":"2025-12-11T04:28:35","modified_gmt":"2025-12-11T04:28:35","slug":"system-performance","status":"publish","type":"page","link":"https:\/\/mrsdprojects.ri.cmu.edu\/2025teamg\/system-performance\/","title":{"rendered":"System Performance"},"content":{"rendered":"\n
\n

Spring System Performance<\/h2>\n\n\n\n

For the Spring Semester, our team set a target to complete the depth estimation, odometry, and mapping subsystems and all system-level requirements associated with those subsystems. The complete list of targeted requirements is shown in the table below.<\/p>\n\n\n\n

Requirement<\/th>Status<\/th>Subsystem<\/th><\/tr><\/thead>
PR.M.1<\/strong> Sense the environment farther than 1m away from itself.
PR.M.2<\/strong> Sense the environment closer than 5m away from itself.<\/td>		Passed<\/td>Depth Estimation<\/td><\/tr>
PR.M.3<\/strong> Perform odometry with a Relative Path Error <2m.<\/td>Passed<\/td>Odometry<\/td><\/tr>
PR.D.3<\/strong> Updates the dense-depth map visualization at minimum 0.2Hz.
PR.D.4<\/strong> Chamfer distance is less than 5m from ground truth point cloud.<\/td>		Passed<\/td>Mapping<\/td><\/tr>
PR.M.6<\/strong> Communicate with an operator with a maximum latency of 500ms.
PR.M.7<\/strong> Communicate with an operator within 25m.<\/td>		Passed<\/td>Networking<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

By the end of spring, the overall system is comprised of the Phoenix Pro drone and the Ground Control Station (GCS). Through a Local Area Network, the drone wirelessly transmits raw sensor feeds as well as the predicted depth, odometry, and global map to the GCS, where operators can view the visualized data in real time. All compute is executed onboard the drone.<\/p>\n\n\n\n

We demonstrated these capabilities at SVD through the following demonstrations:<\/p>\n\n\n\n

Procedure<\/th>Success criteria<\/th>Result<\/th><\/tr><\/thead>
The operator flies the drone around a building that is emitting thick smoke. The drone relays the output of the sensors to the ground control station in real-time.<\/td>Flight verification:<\/em> Continuous flight for greater than 3 minutes
Networking verification: <\/em>Communication Latency <5s (visual verification)<\/td>		Flew outside continuously around smoke for 3 min 40 secs.
Latency on average for all communication was less than 500ms.<\/td><\/tr>
An indoor environment is filled with dense smoke, and the drone is navigated in the indoor environment. The drone localizes itself in the environment and recovers dense metric depth through dense smoke and darkness. The robot then transmits the sensor feeds and output of the algorithms to the ground control station.<\/td>Perception verification:<\/em> Sense the environment from 1m to 5m
Depth estimation verification:<\/em> Dense depth updates should happen at 0.2Hz
Depth estimation verification: <\/em>Chamfer distance less than 20cm.
Odometry verification: <\/em>Relative Path Error < 2m
when compared to Visual SLAM ground truth.
Networking verification: <\/em>Communication Latency <5s (visual verification)<\/td>		Perceived through 0.5m to 20m.
Dense depth updates come in at 0.2Hz.
Chamfer distance less than 20cm.
Relative Path Error was 2cm for easier trajectories and 8cm for the SVD complex indoors trajectory.
Latency on average for all communication was less than 500ms.<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n<\/div>\n<\/div>\n\n\n\n
\n

Fall System Performance<\/h2>\n\n\n\n

For the fall semester, our team set a target to show the system working in both of its use cases. This meant demonstrating the complete mapping, PoI detection, networking and UI, and autonomous flight subsystems and all system-level requirements associated with those subsystems. The complete list of targeted requirements is shown in the table below.<\/p>\n\n\n\n

Although we demo’ed mapping in the spring semester and passed the requirements, one thing we did not achieve was odometry at >10Hz. In order to achieve that, we moved from a learning based VO algorithm to classical VIO. This allowed us to integrate the odometry with the flight controller for autonomy. So in order to regression test, we had to ensure our odometry and mapping performance requirements were still met, which they were. <\/p>\n\n\n\n

Requirement<\/th>Status<\/th>Subsystem<\/th><\/tr><\/thead>
PR.M.4<\/strong> Detect humans in one pose
PR.M.5<\/strong> 80% accuracy given 80% visibility
PR.D.1<\/strong> Localize POI within 5m of ground truth or in the correct room 50% of the time, whichever is applicable<\/td>		Passed<\/td>PoI Detection<\/td><\/tr>
FR.M.4<\/strong> Teleoperation in atleast altitude hold mode and hover in position hold mode
PR.D.2<\/strong> Travel at 10 cm\/s
PR.D.5<\/strong> Any commanded behavior occurs within 2s on the drone<\/td>		Passed<\/td>Autonomy & Flight<\/td><\/tr>
FR.M.6<\/strong> Communicate with an operator
PR.M.6<\/strong> Latency < 500ms
PR.M.7<\/strong> Range \u2264 25m<\/td>		Passed<\/td>Networking & UI<\/td><\/tr>
FR.D.3<\/strong> Compute onboard
PR.D.7<\/strong> 75% or more algorithms must execute on the onboard Orin module<\/td>		Passed<\/td>Real-time Performance<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

We tested these requirements through the following demonstrations:<\/p>\n\n\n\n

Procedure<\/th>Success criteria<\/th>Result<\/th><\/tr><\/thead>
The drone autonomously hovers and lands in thick smoke. The operator can move the drone around without cognitive load. The drone relays the output of the sensors to the ground control station in real-time.<\/td>Flight verification:<\/em> Continuous flight for greater than 3 minutes
Networking verification: <\/em>Communication Latency <5s (visual verification)
Autonomy verification:<\/em> Minimum or no intervention from operator during hold or land<\/td>		– Flew outside continuously around smoke for 3 min 40 secs.
– Latency on average for all communication was less than 500ms.
– No operation intervention was needed<\/td><\/tr>
An indoor environment is filled with dense smoke, and the drone is navigated in the indoor environment. The drone localizes itself in the environment and recovers dense metric depth through dense smoke and darkness. The robot then transmits the sensor feeds and output of the algorithms to the ground control station.<\/td>Perception verification:<\/em> Sense the environment from 1m to 5m
Depth estimation verification:<\/em> Dense depth updates should happen at 0.2Hz
Depth estimation verification: <\/em>Chamfer distance less than 20cm.
Odometry verification: <\/em>Relative Path Error < 2m
when compared to Visual SLAM ground truth.
Networking verification: <\/em>Communication Latency <5s (visual verification)
PoI Detection verification: <\/em>Detect 80% humans in the scene when atleast 80% visible.<\/td>		-Perceived through 0.5m to 20m.
– Dense depth updates come in at 0.2Hz.
– Chamfer distance less than 20cm.
– Relative Path Error was 2cm for complex multi-room FVD path
– Odometry ran at 45 Hz on average.
– Latency on average for all communication was less than 500ms.
– Detected 6 out of 6 humans in the environment even with partial visibility<\/td><\/tr><\/tbody><\/table><\/figure>\n<\/div>\n\n\n\n

Final System Results<\/span><\/strong><\/p>\n\n\n\n

\n
\"\"
Predicted vs Ground truth odometry. Ground truth collected in no smoke with RGB camera. <\/figcaption><\/figure>\n\n\n\n
\"\"
Predicted odometry during simultaneous FFC (camera shutters closed at the same time, i.e., walking blind)<\/figcaption><\/figure>\n<\/div>\n\n\n\n
\"\"
Detecting firefighters during a real residential fire. Data collected using Smores payload<\/figcaption><\/figure>\n\n\n\n
\n
\"\"
Displayed UI with results from all subsystems shown in real-time to the base station through wireless transmission.<\/figcaption><\/figure>\n\n\n\n

Thermal Depth Estimation:<\/strong><\/p>\n<\/div>\n\n\n\n