We have decided to detect hotspots using Yolov4-tiny. We decided to use this over the larger Yolov4 because it runs better on a Jetson, which is what we will be detecting hotspots with in real time. Yolov4-tiny will be used in the Darknet framework.
To capture the specific hotspots of interest — Tables, Light switches, and Door handles — we trained Darknet on a custom dataset consisting of images from these classes on OpenImages and supplemental camera images from our particular testing environment.
The diagram below shows the interaction between the different repositories and services and flow of information to build the custom dataset and train Darknet.
The conversion between the OpenImages and Darknet label formats including abstracting the class name into a class id number and normalizing the bounding box x and y coordinates with the width and height of the image, respectively. An example of an image from OpenImages with the original format and the converted format is shown below.
Creating the labels for the custom images was possible with the Yolo_mark tool. This tool allows hand drawing bounding boxes around hotspots and categorizing them by their hotspot class. The labels and images were then concatenated with the pruned OpenImages dataset before passing them into the darknet training.
The hotspot detection system achieved a mAP of 65% overall on the validation data set after training YOLOv4-tiny on a custom data set combined with a pruned version of the Openimages dataset. Training darknet with YOLOv4 on the same dataset achieved a mAP of 73% overall, with 56% on tables, 63% on door handles, and 100% on light switches. While this improvement is desirable, when we tried to run YOLOv4 on Jetson it performed so slowly it was nonfunctional. This tradeoff lead us to choose the slightly worse performing in mAP, but better performing in real time, YOLOv4-tiny. Since tables was the lowest mAP, we will continue to add to and improve the dataset to contain more tables.
The integration of the Detection and Estimation subsystems can be seen in the diagram below. The main interface is a public repository, the darknet_ros node.
Below is a snapshot of the real time detection and estimation of the light switch and door handle hotspots over the Intel Real-sense.
Click here to see a video of the real time detection and estimation integration