This section summarizes the Flying Hand’s performance against the system-level requirements established at the start of the project, with results drawn from the Spring Validation Demonstration (SVD) and the SVD Encore. Performance is reported across the three subsystems that drive the overall mission capability: Localization, Manipulation, and Full-Pipeline Simulation.
Performance Requirements and Metrics
The system was evaluated against the mandatory performance metrics derived from our functional requirements:
| ID | Functional Requirement | Metric ID | Performance Target | Result |
|---|---|---|---|---|
| MF1 | Sense the surrounding environment | MP1 | Sensing range 0.5–4 m | Met |
| MF2 | Localize own pose relative to the environment | MP2 | Position error ≤ 10 cm under nominal conditions | Met (near-wall hover); Partially met (open-area waypoint) |
| MF3 | Allow operator to manually pilot to an Area of Interest | MP3 | Manual flight speeds up to 1 m/s | Met |
| MF5 | Manipulate end-effector to establish contact with the POI | MP4 | Detect POI from distance ≥ 2.5 m | Met (in simulation) |
| MF6 | Maintain stable end-effector contact during NDT | MP5 | End-effector positioning accuracy within ±0.15 m during contact-based NDT | Met (in simulation) |
| MF10 | Provide real-time visual feedback to the operator | — | Live video and telemetry to ground station | Met |
1. Localization (VIO) Performance
The localization subsystem uses a downward/forward-tilted Intel RealSense D456 camera feeding Isaac ROS Visual SLAM (cuVSLAM) on an NVIDIA Jetson Orin Nano, with the resulting odometry fused into PX4’s EKF2 via MAVROS as an external vision source. Two flight regimes were evaluated.
Near-Wall Position Hold
- Setup: ~0.3 m standoff from a flat wall, ~1.5 m flight height.
- Result: The drone maintained a stable hover for the full 30-second duration of the test, with arm-end keyframes captured during hover confirming positional stability suitable for contact-based NDT.
- Conclusion: Near-wall position hold met the MP2 requirement.
Open-Area Waypoint Navigation
- Setup: Autonomous waypoint navigation followed by an automatic landing target.
- Result: The drone completed waypoint navigation and landed automatically within 0.5 m of the target. However, in autonomous mode, PX4’s EKF2 did not consistently prioritize VIO over the IMU accelerometer, leading to localization drift and degraded waypoint tracking. Manual teleoperation was required mid-flight to recover.
- Conclusion: Functional capability demonstrated; further EKF2 tuning is required for fully autonomous operation. This is the primary VIO work item carried into the fall semester.
VIO Benchmark Against RTK-GNSS Ground Truth
To quantify VIO accuracy, a partial trajectory segment was logged simultaneously with VIO odometry and an RTK-GNSS module serving as ground truth.
| Metric | RTK Ground Truth | VIO | Error |
|---|---|---|---|
| Trajectory length | 24.85 m | 25.11 m | +1.0% |
| Heading angle | −48.9° | 165.5° | −145.6° |
| Straightness (RMS from line) | — | 0.270 m | — |
VIO trajectory length matched ground truth to within 1%, confirming good scale recovery. The large heading offset is an alignment-frame difference rather than a tracking failure, since VIO and RTK report yaw in different reference frames; absolute heading validation was constrained by site limitations (see below). Note that ground truth coverage was limited because magnetometer readings at the test site were unreliable due to heavy magnetic interference, and low satellite count from RF interference prevented the RTK module from achieving a fix over the full flight.
2. Manipulation Performance
The manipulation subsystem combines the fully-actuated hexarotor platform with the force/torque sensor and Wrench Controller.
- The fully-actuated drone successfully made contact with flat target surfaces under wind speeds up to 2 m/s with gusts of 5 m/s.
- The force sensor accurately measured contact force at the 5 N target level.
- The Wrench Controller computed the desired thrust output corresponding to measured contact force, closing the loop on contact-based manipulation.
- Conclusion: All manipulation performance criteria for SVD were met.
3. Full-Pipeline Simulation Performance
The end-to-end mission was validated in Gazebo simulation, integrating the fully-actuated platform, F/T sensor, camera, target localization, and visual servoing.
| Capability | Target | Simulated Result |
|---|---|---|
| Effective range of visual servo controller | > 2.5 m | Met |
| Arm-end contact error | < 0.15 m | Met |
| Force-convergence time after contact | < 3 s | Met |
| Constant force application | 5 ± 1 N | Met |
| Duration of force application | ≥ 15 s | Met (continuous 5 N force applied for > 10 s; sustained behavior confirmed) |
In simulation, the drone autonomously took off, detected the POI, used visual servoing to guide its approach, and upon contact the Wrench Controller applied a continuous 5 N force, confirming overall system readiness for hardware integration in the fall semester.
SVD Conclusion
- Manipulation: All SVD performance criteria met on hardware.
- Localization: Near-wall position hold met requirements; open-area autonomous waypoint navigation requires further EKF2 tuning.
- Full Pipeline: End-to-end behavior validated in simulation; hardware integration is the focus of the fall semester.
Lessons Learned and Constraints Affecting Performance Evaluation
- Test site interference: Magnetometer readings were unreliable due to heavy magnetic interference, and low satellite count (likely from RF interference) prevented GPS/RTK from functioning fully, limiting ground-truth trajectory validation to a partial segment.
- Hardware integration overhead: Modules assumed to be plug-and-play (optical flow, RTK module, F/T sensor alternatives) each required substantial troubleshooting and, in some cases, replacement.
- Single-owner subsystems: Each subsystem being primarily owned by one team member created blind spots; broader cross-coverage is planned for the fall.