{"id":355,"date":"2021-03-03T19:50:32","date_gmt":"2021-03-03T19:50:32","guid":{"rendered":"http:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/?page_id=355"},"modified":"2021-05-11T21:35:33","modified_gmt":"2021-05-11T21:35:33","slug":"hotspot-detection","status":"publish","type":"page","link":"https:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/hotspot-detection\/","title":{"rendered":"Hotspot Detection"},"content":{"rendered":"<p>We have decided to detect hotspots using <a href=\"https:\/\/blog.roboflow.com\/train-yolov4-tiny-on-custom-data-lighting-fast-detection\/\">Yolov4-tiny<\/a>. We decided to use this over the larger <a href=\"https:\/\/blog.roboflow.com\/training-yolov4-on-a-custom-dataset\/\">Yolov4<\/a> because it runs better on a Jetson, which is what we will be detecting hotspots with in real time. Yolov4-tiny will be used in the <a href=\"https:\/\/github.com\/AlexeyAB\/darknet\">Darknet<\/a> framework.&nbsp;<\/p>\n<p>To capture the specific hotspots of interest &#8212; Tables, Light switches, and Door handles &#8212; we trained Darknet on a custom dataset consisting of images from these classes on OpenImages and supplemental camera images from our particular testing environment.<\/p>\n<p>The diagram below shows the interaction between the different repositories and services and flow of information to build the custom dataset and train Darknet.&nbsp;<\/p>\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1010\" height=\"406\" src=\"https:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/wp-content\/uploads\/sites\/50\/2021\/05\/dataset_architecture.png\" alt=\"\" class=\"wp-image-728\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/wp-content\/uploads\/sites\/50\/2021\/05\/dataset_architecture.png 1010w, https:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/wp-content\/uploads\/sites\/50\/2021\/05\/dataset_architecture-300x121.png 300w, https:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/wp-content\/uploads\/sites\/50\/2021\/05\/dataset_architecture-768x309.png 768w\" sizes=\"auto, (max-width: 1010px) 100vw, 1010px\" \/><figcaption>Detection Custom Dataset and Training Architecture<\/figcaption><\/figure>\n\n\n\n<p>The conversion between the OpenImages and Darknet label formats including abstracting the class name into a class id number and normalizing the bounding box x and y coordinates with the width and height of the image, respectively. An example of an image from OpenImages with the original format and the converted format is shown below.<\/p>\n\n\n\n<figure class=\"wp-block-gallery columns-1 is-cropped wp-block-gallery-1 is-layout-flex wp-block-gallery-is-layout-flex\"><ul class=\"blocks-gallery-grid\"><li class=\"blocks-gallery-item\"><figure><img loading=\"lazy\" decoding=\"async\" width=\"564\" height=\"758\" src=\"https:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/wp-content\/uploads\/sites\/50\/2021\/03\/Screen-Shot-2021-03-17-at-5.18.19-PM.png\" alt=\"\" data-id=\"493\" class=\"wp-image-493\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/wp-content\/uploads\/sites\/50\/2021\/03\/Screen-Shot-2021-03-17-at-5.18.19-PM.png 564w, https:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/wp-content\/uploads\/sites\/50\/2021\/03\/Screen-Shot-2021-03-17-at-5.18.19-PM-223x300.png 223w\" sizes=\"auto, (max-width: 564px) 100vw, 564px\" \/><\/figure><\/li><\/ul><figcaption class=\"blocks-gallery-caption\">Example image from the OpenImages dataset with a light switch and door handle visible<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"860\" height=\"54\" src=\"https:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/wp-content\/uploads\/sites\/50\/2021\/03\/Screen-Shot-2021-03-17-at-4.09.29-PM-2.png\" alt=\"\" class=\"wp-image-496\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/wp-content\/uploads\/sites\/50\/2021\/03\/Screen-Shot-2021-03-17-at-4.09.29-PM-2.png 860w, https:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/wp-content\/uploads\/sites\/50\/2021\/03\/Screen-Shot-2021-03-17-at-4.09.29-PM-2-300x19.png 300w, https:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/wp-content\/uploads\/sites\/50\/2021\/03\/Screen-Shot-2021-03-17-at-4.09.29-PM-2-768x48.png 768w\" sizes=\"auto, (max-width: 860px) 100vw, 860px\" \/><figcaption>OpenImages Formatted Label: The order of the text indicates (from left to right) Class Name, Left X, Top Y, Right X, Bottom Y, where Left X, Top Y, Right X, Bottom Y are the pixel coordinates of the bounding box.<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/wp-content\/uploads\/sites\/50\/2021\/03\/Screen-Shot-2021-03-17-at-4.09.05-PM.png\" alt=\"\" class=\"wp-image-497\" width=\"569\" height=\"52\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/wp-content\/uploads\/sites\/50\/2021\/03\/Screen-Shot-2021-03-17-at-4.09.05-PM.png 588w, https:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/wp-content\/uploads\/sites\/50\/2021\/03\/Screen-Shot-2021-03-17-at-4.09.05-PM-300x28.png 300w\" sizes=\"auto, (max-width: 569px) 100vw, 569px\" \/><figcaption>Darknet Formatted Label: The order of the text indicates (from left to right) Class ID, LeftX\/image width, TopY\/image height, RightX\/image width, BottomY\/image height, where LeftX, TopY, RightX, BottomY are the pixel coordinates of the bounding box. Shown here is a light switch classified with class id 0, and a door handle classified with class id 1.<\/figcaption><\/figure>\n\n\n\n<p>Creating the labels for the custom images was possible with the <a href=\"https:\/\/github.com\/AlexeyAB\/Yolo_mark\">Yolo_mark<\/a> tool. This tool allows hand drawing bounding boxes around hotspots and categorizing them by their hotspot class. The labels and images were then concatenated with the pruned OpenImages dataset before passing them into the darknet training. <\/p>\n\n\n\n<p>The hotspot detection system achieved a mAP of 65% overall on the validation data set after training YOLOv4-tiny on a custom data set combined with a pruned version of the Openimages dataset. Training darknet with YOLOv4 on the same dataset achieved a mAP of 73% overall, with 56% on tables, 63% on door handles, and 100% on light switches. While this improvement is desirable, when we tried to run YOLOv4 on Jetson it performed so slowly it was nonfunctional. This tradeoff lead us to choose the slightly worse performing in mAP, but better performing in real time, YOLOv4-tiny. Since tables was the lowest mAP, we will continue to add to and improve the dataset to contain more tables.<\/p>\n\n\n\n<p>The integration of the Detection and Estimation subsystems can be seen in the diagram below. The main interface is a public repository, the darknet_ros node.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"613\" height=\"267\" src=\"https:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/wp-content\/uploads\/sites\/50\/2021\/05\/detectiontoestimation.png\" alt=\"\" class=\"wp-image-729\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/wp-content\/uploads\/sites\/50\/2021\/05\/detectiontoestimation.png 613w, https:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/wp-content\/uploads\/sites\/50\/2021\/05\/detectiontoestimation-300x131.png 300w\" sizes=\"auto, (max-width: 613px) 100vw, 613px\" \/><figcaption>Real-time Detection of Hotspots to Bounding Boxes Architecture<\/figcaption><\/figure>\n\n\n\n<p>Below is a snapshot of the real time detection and estimation of the light switch and door handle hotspots over the Intel Real-sense. <\/p>\n\n\n\n<figure class=\"wp-block-gallery columns-2 is-cropped wp-block-gallery-2 is-layout-flex wp-block-gallery-is-layout-flex\"><ul class=\"blocks-gallery-grid\"><li class=\"blocks-gallery-item\"><figure><img loading=\"lazy\" decoding=\"async\" width=\"438\" height=\"248\" src=\"https:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/wp-content\/uploads\/sites\/50\/2021\/04\/2.png\" alt=\"\" data-id=\"598\" class=\"wp-image-598\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/wp-content\/uploads\/sites\/50\/2021\/04\/2.png 438w, https:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/wp-content\/uploads\/sites\/50\/2021\/04\/2-300x170.png 300w\" sizes=\"auto, (max-width: 438px) 100vw, 438px\" \/><\/figure><\/li><li class=\"blocks-gallery-item\"><figure><img loading=\"lazy\" decoding=\"async\" width=\"474\" height=\"300\" src=\"https:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/wp-content\/uploads\/sites\/50\/2021\/04\/Screen-Shot-2021-04-14-at-10.37.32-AM.png\" alt=\"\" data-id=\"599\" data-full-url=\"https:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/wp-content\/uploads\/sites\/50\/2021\/04\/Screen-Shot-2021-04-14-at-10.37.32-AM.png\" data-link=\"https:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/hotspot-detection\/screen-shot-2021-04-14-at-10-37-32-am\/\" class=\"wp-image-599\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/wp-content\/uploads\/sites\/50\/2021\/04\/Screen-Shot-2021-04-14-at-10.37.32-AM.png 474w, https:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/wp-content\/uploads\/sites\/50\/2021\/04\/Screen-Shot-2021-04-14-at-10.37.32-AM-300x190.png 300w\" sizes=\"auto, (max-width: 474px) 100vw, 474px\" \/><\/figure><\/li><\/ul><figcaption class=\"blocks-gallery-caption\">(Left) Intel RealSense frame with bounding box visualization of hotspot detection and confidence level. (Right) RViz visualization of hotspot estimation in relation to the robot base<\/figcaption><\/figure>\n\n\n\n<p><a href=\"https:\/\/drive.google.com\/file\/d\/1knCAS_0068AaRUzGw8rLvXL7Y4U2_fqW\/view?usp=sharing\" data-type=\"URL\" data-id=\"https:\/\/drive.google.com\/file\/d\/1knCAS_0068AaRUzGw8rLvXL7Y4U2_fqW\/view?usp=sharing\">Click here to see a video of the real time detection and estimation integration<\/a><\/p>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>We have decided to detect hotspots using Yolov4-tiny. We decided to use this over the larger Yolov4 because it runs better on a Jetson, which is what we will be detecting hotspots with in real time. Yolov4-tiny will be used in the Darknet framework.&nbsp; To capture the specific hotspots of interest &#8212; Tables, Light switches,&hellip;&nbsp;<a href=\"https:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/hotspot-detection\/\" rel=\"bookmark\"><span class=\"screen-reader-text\">Hotspot Detection<\/span><\/a><\/p>\n","protected":false},"author":230,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"template-fullwidth.php","meta":{"neve_meta_sidebar":"","neve_meta_container":"","neve_meta_enable_content_width":"","neve_meta_content_width":0,"neve_meta_title_alignment":"","neve_meta_author_avatar":"","neve_post_elements_order":"","neve_meta_disable_header":"","neve_meta_disable_footer":"","neve_meta_disable_title":"","footnotes":""},"class_list":["post-355","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/wp-json\/wp\/v2\/pages\/355","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/wp-json\/wp\/v2\/users\/230"}],"replies":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/wp-json\/wp\/v2\/comments?post=355"}],"version-history":[{"count":16,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/wp-json\/wp\/v2\/pages\/355\/revisions"}],"predecessor-version":[{"id":741,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/wp-json\/wp\/v2\/pages\/355\/revisions\/741"}],"wp:attachment":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2021teama\/wp-json\/wp\/v2\/media?parent=355"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}