Full System Spring Semester Implementation
- This is the overview of the Spring Progress.
- Please refer to subsystems for subsystem details.
Spring System Diagram
FireSense is composed of two cooperating components: a Payload carried into the operational environment and a Command Station located outside the hazard zone. The two communicate wirelessly over a 915 MHz HaLow link, allowing the operator to move untethered while streaming sensor data back for analysis. The Payload handles real-time data acquisition and the SLAM frontend, while the Command Station handles global optimization, loop closure, and visualization for the Fire Chief.
Below is a brief overview of each subsystem. For detailed technical descriptions, please refer to the dedicated subsystem pages.
Hardware – Payload v1.2
The current payload revision adopts a handle-style form factor optimized for handheld use in firefighting scenarios. Compared to v1.1, it has a smaller footprint, improved internal wiring, and increased mechanical durability for faster motion. The modular enclosure simplifies maintenance and integrates all required sensors—including the FLIR Boson+ thermal cameras—into a single unit. Thermal management is handled passively through louver and honeycomb features machined into the housing.
The system integrates a stereo thermal camera (two FLIR Boson+ sensors) with a ZED-X sensor for inertial measurements. The ZED acts as the master device providing a 30 Hz hardware trigger; the Boson+ cameras run in slave mode for synchronization, outputting effective 15 Hz frames. Stereo calibration is performed using a heated 1m × 1m calibration board for thermal contrast, and Supplemental Flat Field Correction (SFFC) is applied after the system reaches stable operating temperature to ensure accurate thermal readings.
Hardware – Communication
The system uses Wi-Fi HaLow at 915 MHz for wireless communication, providing better wall penetration and longer range than conventional 2.4 GHz Wi-Fi. The setup consists of a HaLow router at the Command Station and HaLow USB dongles attached to each payload via Ethernet in bridge mode. The middleware was migrated from FastDDS multicast to CycloneDDS with a Zenoh bridge using unicast, reducing average incoming traffic by 81% and maximum outgoing traffic by 99.7%, which dramatically improves stability for multi-payload C-SLAM.
Hardware – PCB Design
A custom PCB designed in Altium serves as the payload’s power distribution and debug board. It steps 18V down to stable 12.26V and 5.03V rails. During integration, the buck converters experienced a critical failure—likely caused by LC resonant ringing from the Makita battery connection spiking input voltage. The next revision (PCB 2.0) will add input-rail bulk capacitance for surge protection along with dimensional updates for Payload 2.0 integration.
Software – SLAM Frontend
The SLAM frontend processes stereo thermal images and IMU data to produce real-time ego-motion estimates. Raw 14-bit thermal images are normalized, contrast-enhanced (CLAHE), rectified, and corrected via SFFC before being passed to MAC-VIO (Metric-Aware Covariance Visual-Inertial Odometry), a loosely coupled visual-inertial framework built on the GTSAM iSAM2 optimizer. We transitioned MAC-VO’s backend optimization from PyPose to a GTSAM-based factor graph framework to improve computational efficiency and scalability, and defined a unified Keyframe Data Structure to standardize communication between the frontend and backend with clean subsystem interfaces.
To maximize runtime efficiency on the Jetson AGX Orin, we implemented the NVIDIA NITROS pipeline, enabling zero-copy GPU transport between ROS 2 nodes. This eliminated unnecessary CPU–GPU memory transfers and allowed direct conversion of accelerated image streams into PyTorch tensors for inference. The frontend currently runs onboard with TensorRT-accelerated inference at approximately 4 Hz, achieving a relative translation error (RTE) of 0.0066 m, and publishes odometry and 3D keypoints in real time.
Software – SLAM Backend
The Command Station runs the SLAM backend, fusing sequential odometry with appearance-based loop closure in a single GTSAM pose graph. Each keyframe is assigned a global descriptor distilled from ViT-B/14 with SALAD pooling (adapted for thermal imagery via AnyThermal). Loop hypotheses are generated through cosine similarity using a FAISS index, then verified geometrically. Incremental iSAM2 updates the graph in real time, with periodic Levenberg–Marquardt optimization for global consistency.
In parallel, we prototyped ORB-based loop closure detection on collected datasets and evaluated strategies for handling feature-poor, fast-motion conditions typical of fire scenes. The Spring Validation Demo at ACFA achieved 0.18 m ATE after loop closure.
System Design & Integration
The overall system is organized into three logical layers: Payload Frontend, Communication Layer, and Centralized SLAM Backend. The payload frontend performs sensor acquisition, preprocessing, visual odometry, and IMU integration. Keyframes and associated constraints are transmitted through the wireless bridge to the Command Station, where the GTSAM-based backend constructs and optimizes the global factor graph.
By structuring the system around modular subsystems with explicit data contracts—including loop closure detection, keyframe selection logic, and IMU preintegration strategies—integration risk was reduced and parallel development across hardware, frontend perception, and backend optimization was enabled. At the conclusion of the spring term, all subsystems are functionally integrated: hardware is assembled, the SLAM pipeline is operational on embedded compute, and defined interfaces support continued refinement heading into the fall semester.