2025 Team G

System Implementation

Odometry

Benchmarked Two Odometry Algorithms:

  • Robust Visual Thermal Inertial Odometry (ROVTIO)
  • Multi-Spectral Odometry (MSO)

Benchmarks with real-world data show these algorithms to be robust and reliable in the face of perceptual degradations such as smoke.

A sample trajectory captured (and the corresponding ground-truth) in the presence of smoke is shown.

Dense Depth Estimation from Thermal Cameras

Our current depth estimation subsystem uses MoGe to estimate relative depth and MadPose to compute metric depth. The operational workflow is as follows:

  • Time Synchronization: Captured images from the stereo thermal setup are time-synchronized using a hardware pulse.
  • Preprocessing: The 16-bit depth data is preprocessed to optimize feature extraction.
  • Relative Depth Estimation: The MoGe algorithm generates relative depth maps for both the left and right thermal cameras.
  • Metric Depth Computation: The MadPose optimizer computes metric depth using the relative depth outputs from both cameras, leveraging camera extrinsic parameters.
Relative Depth Estimation
Solving for Metric depth

Hardware Design

Mounts are being designed for the two thermal cameras, while keeping in mind the FOV of the thermal cameras and protection from crashes.

Our design, shown below, places the cameras within the convex hull of the drone’s body, thus making it safer during crashes as it will not be first to strike the ground. The design also keeps the propellers outside of the FOV of the cameras. The mounting positions ensure that the thermal camera mounts are single fault tolerant, in the sense that every degree of freedom is secured in at least two mounting directions in case any one of them become loose.

Sensor payload mounts

Considering rapid iteration as we decide on our sensor payload and drone design, major subassemblies of the drone are designed for not only robustness but modularity and ease of manufacturing and assembly.

Orin Subassembly
Battery Subassembly

The integrated hardware for the drone includes:

  • Nvidia AGX Orin for compute,
  • 2 Flir Boson cameras for thermal vision,
  • Intel Realsense D456 for RGBD vision,
  • LiPo battery for power,
  • Garmin Lidar Lite v3 for altitude sensing, and
  • Pixracer Pro for flight controller,

All built on a Hexsoon EDU-650 airframe.

Integrated system hardware design

Electrical Design

The drone now boasts 2 Flir Boson Cameras, an Intel RealSense, Epson IMU, and the Nvidia Jetson AGX Orin. Interfacing and synchronizing the sensor stack with the compute and the flight controller is complicated and has to be meticulously designed. We have achieved sensor interfacing and integration and we document all the components of our subsystems with wiring diagrams and takeaways. Below is an example for how our sensors interface with the Nvidia Orin’s GPIO pins.

Wiring diagram of sensors interfacing with Nvidia Jetson AGX Orin using the GPIO pins

Additionally, we have also encountered custom breakout boards designed by alumni which there is no documentation on when dealing with the FLIR Thermal cameras and the Epson IMU G365. Understanding their work and documenting it allows us to debug issues we are facing. Currently, the IMU gets stuck on initialization and does not consistently publish messages. Looking at the breakout board designed we were able to eliminate power supply as a cause for the problem.

The board file of the breakout designed by a CMU alumni that we reverse engjneered for debugging IMU reliability.

Networking

Hardware

We have at our disposal:

  • The Orin’s WiFi
  • GCS Air Unit (on board drone)
  • GCS Ground Unit (on ground)
  • Control Computer

Architecture

We have two alternative networking solutions:

Orin Wifi

The Orin publishes a WiFi Hotspot using it’s own hardware and antennas attached to it.

This approach is unstable and fragile, wifi strength drops quickly with distance.

We use this for quick and dirty communication, where we expect disconnects.

GCS Air to Ground LAN

The GCS enables us to bridge a LAN connection between the GCS ground and air units.

We extend this to the Control Computer and the Orin, respectively.

We set routes so the Orin and Control Computer exist on the same subnet and route messages via the Air Unit LAN so they can be seen at both ends.

Video Transmission

Sending video frame-by-frame is inefficient.

We encode the video into a stream using h.264 and then send transmitted data over the network.

This results in lesser data being sent, as encoded video avoids transmitting redundant information.

At the receiver, we decode the stream and publish it to an Image topic, so that ROS tools like RViz can view the images.

Low-Level Sensor integration

Thermal Cameras: The drone uses 2 FLIR Boson cameras giving us 640×512 resolution. We wrote our own custom driver using V4L2 to allow for quick processing and efficient memory management. The driver also allows the thermal camera to be set to disabled, master, or slave mode and this changes the timing characteristics of the cameras. We have also developed a thermal preprocessing method that takes into account thermal-sensor noise adjustment using FFC, gain correction, histogram equalization, and noise estimation.

RealSense 465: This is to measure ground truth and provide some depth estimation when flying outdoors or in environment where visual features can be leveraged. The RealSense has 2 imagers, 1 color sensor and 1 IR projector along with an internal IMU.

IMU: The drone relies on 2 Inertial Measurement Units – Epson G365 external IMU and one that is included with the RealSense 465. The team has worked towards improving the Epson IMU driver code as by-product of testing and debugging issues with it.

Logistics and Misc.

Two members of the team have obtained Part-107 drone licenses in order to pilot drones.

Arranged for testing sites at two locations:

  • NREC Drone Cage
  • Fire Academy