System Details
Subsystem Details
Problem Description
Residential building fires pose a severe threat to public safety, resulting in thousands of deaths and over $11 billion in financial losses in 2023 alone, as shown below.
During these emergencies, the first few minutes (primary search) are the most critical in minimizing casualties and damages. Firefighters conducting primary searches in smoke filled rooms face critical operational challenges, such as missing directional cues and a sensing range limited to their immediate physical reach.
Furthermore, Fire Chiefs coordinating efforts outside the hazard zone often lack real-time information regarding the interior layout and the precise location of their team members, significantly hampering effective coordination and decision-making.
To address these critical safety gaps, FireSense is a modular sensing system designed to aid firefighters by establishing real-time operational awareness during structural fires.
Unlike traditional robotic solutions that rely on independent platforms, this project utilizes a human-centric approach where wearable payloads are carried directly by firefighters.
This design eliminates the friction of deploying separate ground or aerial robots, allowing the system to seamlessly integrate into existing firefighter workflows while extending their situational awareness beyond physical barriers.
The system fulfills the distinct operational needs of two primary user groups: deployed firefighters and the Fire Chief.
For the Fire Chief at the Command Station, the system provides a centralized interface that displays a collaboratively generated 3D map of the building in real-time.
By tracking the live positions of firefighters and identifying the locations of victims, the system enables the fire chief to make faster tactical decisions, vector rescue teams efficiently to points of interest, and coordinate rapid interventions based on live data.
Simultaneously, the system functions as a digital partner for the firefighter, automatically detecting and mapping the environment to locate victims, thereby improving overall safety and speeding up the primary search process.
Use Case
Upon arrival at a residential structure fire, the Fire Chief performs a 360° exterior inspection to identify entry points, hazards, and structural risks. Once the primary search begins, two firefighters enter the smoke-filled building equipped with the FireSense wearable payload system, integrated with their air-tank backpack.
Wearable Payload Deployment
The FireSense payload is mounted on the firefighter’s shoulder and upper back, maintaining balance and minimizing interference with standard gear and movement. The heavy compute unit (Jetson AGX Orin) and power supply sit on the back, with wired connections running to the lighter sensors on the shoulder. As firefighters advance through hallways, staircases, and rooms under zero-visibility conditions, the payload continuously senses the environment using thermal stereo cameras and radar. Even in hot smoke where vision is limited to arm’s length, the system captures structural layout, obstacles, and potential victims.
Edge Processing & Wireless Transmission
Instead of transmitting heavy raw data streams, the wearable payload processes sensor inputs locally on the AGX Orin to generate lightweight SLAM constraints and Point-of-Interest (POI) detections. These compact outputs are transmitted wirelessly through walls via a 915 MHz Wi-Fi HaLow link to the Command Station located safely outside the hazard zone. This edge-processing approach reduces bandwidth requirements and allows the system to tolerate intermittent connectivity through dense walls.Edge Processing & Wireless Transmission
Command Station & Collaborative SLAM
At the Command Station, a centralized Collaborative SLAM (C-SLAM) backend fuses data from all deployed payloads to construct a consistent, real-time 3D map of the building interior. The interface provides the Fire Chief with operational awareness that traditionally does not exist during interior operations, displaying:
- A live global 3D map of the structure
- The real-time positions of all active firefighters
- Detected Points of Interest (e.g., victims, exits, hazards)
- Navigation path recommendations to the nearest egress or victim location
Tactical Decision-Making
Using this real-time operational awareness, the commander can coordinate the nearest firefighter or dispatch a rescue team to the exact location provided by the live map. In the event of a critical situation such as a Mayday call, the instant 3D mapping and 2D annotated floor plan allow the incident commander to make faster tactical decisions—dispatching rescue teams to a precise location rather than relying on rough verbal descriptions.
