Functional Architecture
The functional architecture of the PhoeniX firefighting system has been depicted in Figure below. It captures all the functionalities we derived from the functional requirements and the objectives tree. The operation of the system begins when an operator (person/system) triggers a start signal to the system which will be in the form of the approximate GPS location of the building under fire. Given this start signal the UAVs take off and the AGV drives off following a trajectory precomputed by the AGV onboard and sent to the UAVs. While they travel towards the fire location, the systems start to map the environment while avoiding any obstacles. This map will be used by the system to plan path from point A to point B.While the systems are exploring the environment, they will also keep on checking for any potential fire locations by using the fire detection subsystem. Once a system identifies a location with fire, it will inform other systems by adding the location of the fire in a shared database. If there are fire extinguishing tasks pending in the database, a intelligent task assigner/scheduler will command a system with enough extinguishing resources to navigate to the fire location and extinguish fire.
So, once the systems will receive the co-ordinates of the fire location and they shall autonomously navigate in the environment while avoiding obstacles and now once they reach the proximity of fire, they will orient themselves in an appropriate position to extinguish fire. Now the system will deploy the extinguishing payload using some strategy and update the database
CyberPhyical Architecture
Below figure shows our Cyberphysical architecture which maps the flow of data and energy between components and subsystems based on the results on our trade studies. Input: GPS coordinates of the target location is being sent to the System.- UAV/AGV System: Based on the trade studies, we have finalized combination of Stereocamera, IMU, GPS as mapping sensors on UAV and an additional 2D LiDAR is added on AGV for better localization. Different mobile robots communicate via Wi-Fi link.
- Exploration Mode: Mapping sensors will generate an occupancy grid map for obstacle avoidance and planning local trajectory. Each mobile robot will run YOLO fire detection classifier for fire detection.
- Scheduler: Each mobile robot will detect fire and update the shared database. Based on the fire location in shared database, Scheduler will assign fire extinguishing to different robots based on their locations from the fire.
- Extinguishing Mode: Based on the global fire location, mobile robots will do visual servoing towards the assigned fire location, monitor and extinguish fire.
Subsystem Description
- Hardware Subsystems
- UAV Subsystem While the design of the base UAV (tilted rotor hexacopter) is provided by the sponsors (Sebastian Scherer and Oliver Kroemer), the firefighting task necessitates to modify existing design to integrate thermal and stereo camera sensors as well as mechanism for attaching and deploying payload (extinguishing material) at the target. For low level control in deploying mechanism, microcontroller and actuators would be required and for processing thermal image data, single board computer (Nvidia Jetson) would be required. Integrating these modules would requires us to redesign power distribution board as well. UAV hardware subsystem will handle modifying existing design to handle above mentioned requirements. UAV hardware system will include assembling various parts to build the UAV (under guidance of Air Lab, CMU). It will also provide basic user interface along with kill switch. The below drone shows an early prototype of the UAV in December 2018.
- AGV Subsystem Pre-built base AGV system (Husky) is provided by the sponsors but like UAV system certain modifications are required to do firefighting task. Major difference compare to UAV is that there are no power constraints or significant payload constraints which would allow us to integrate LiDAR sensors for robust collision avoidance along with stereo and thermal camera sensors and extra extinguishing material. Design for attaching and deploying extinguishing material at target would be quite different and would require more Degree of Freedom since yaw and height of AGV can’t be changed. Single Board Computer as well as Wifi Base station would also needs to be integrated. AGV Subsystem will take of designing and physically integrating parts together. It will also provide basic user interface with kill switch.
- Software Subsystems There are some key differences between UAV and AGV system but for the most part software subsystem developed for one can be used in other with minor modification. Considering time and resources available for MRSD project, it doesn’t make sense to create separate software subsystem for UAV and AGV. This implies that all the software solutions would be generic such that it can work on both UAV and AGV but may not be optimum individually.
- Simultaneous Localization and Mapping (SLAM) Subsystem Since exact 3D map of the outdoor or indoor of the environment is not provided, We need to create 3D map of the environment which would allow Path Planning Subsystem to avoid collision and Navigation Control Subsystem to generate better control signals. While distributed multi-agent SLAM would lead to better Map and Localization, Our UAV/AGV system will not do distributed SLAM due to added complexity and reliance on high communication bandwidth. UAV and AGV do communicate about detected fire location but both can act and operate independently and are not handicapped in case of communication failure. SLAM Subsystem will provide “real time” map construction and localization in the environment using stereo camera sensor and will be computationally efficient to run on UAV as well.
- Path Planning Subsystem PhoeniX firefighting system is essentially a heterogeneous multiagent system. While approximate idea about location of building would be known to the system, exact fire location in the building is not known. UAV and AGV both needs to collaboratively search for fire in the environment efficiently i.e. we don’t want both system to search overlapping area. Path planning subsystem will provide both global path planning as well as local path planning. Global planning will include high level exploration path for both UAV, AGV in such way that they jointly achieve high coverage while exploration (Frontier based exploration Algorithm). Location planner involves taking cues from global planner and point cloud map from mapping subsystem to produce path that detours obstacles location to avoid any collision. (RRT Algorithm) Path planning subsystem will provide its output of desired trajectory as way points to Navigation Control subsystem.
- Navigation Control Subsystem Based on mode of operation, Navigation Control Subsystem takes input from Path Planning Subsystem or Fire Localization Subsystem and will create control signals to go to desired waypoint or follow desired trajectory. These control signals would be generated in closed loop fashion. Navigation controller would take localization data from SLAM Subsystem and will merge it with GPS and IMU data to get accurate localization (feedback) to generate control signals that corrects deviation from desired trajectory. Navigation control subsystem will communicate directly with low level motor controllers.
- Collaboration Subsystem Collaboration Subsystem will provide communication link to transfer vital information such as active fire location, other system’s resources status. This information can be extremely useful for high level decision making of what to do. (Explore, Extinguish or Return) Communication link would enable our system to fight the fire in collaborative manner. Using this communication, UAV/AGV can ask for help in extinguishing fire if fire is too large. Collaboration Subsystem will create a shared database containing active fire location and would assign tasks based on priority queue.
- Fire Detection and Localization Subsystem As the name suggests, goal of this subsystem is to detect fire from the thermal image. To accurately create bounding box around fire in thermal image, image classifier needs to be trained. Thermal Image Dataset needs to be procured and darknet needs to be trained. Once classifier is available, it needs to be implemented in single board computer of UAV and AGV. Once integrated, Fire detection Subsystem keeps looking for fire and when it detects fire and it looks at output of SLAM Subsystem, to approximate location of fire in global map. Finally, Fire Detection and Location Subsystem shares this fire location to Extinguishing Fire Subsystem as well as Collaboration system so that other system can also become aware of that fire.
- Fire Extinguishing Subsystem Once extinguishing task is assigned after detecting the fire. Fire extinguishing subsystem becomes active. In first step, it orients UAV/AGV with respect to fire by providing waypoints to Navigation Control Subsystem using Visual Servoing method. E.g. if system is far away from the fire, waypoints that leads the system to fire are provided. In second step, it provides control signals to microcontroller which in turn activates the actuators of extinguishing deploying mechanism to point it directly at fire. Finally it gives signal to deploy the material and provides alert to Navigation Control Subsystem so that it can handle recoil and payload change. It keeps monitoring the fire status using thermal image data to stop deployment if fire is extinguished. It also keeps track of how much extinguishing material is available using load sensor.
We quickly assembled a new drone for our application with AirLabs assistance and the below picture shows our drone which was used for SVD and throughout the Spring '19 semester.
