{"id":261,"date":"2016-10-21T22:22:02","date_gmt":"2016-10-21T22:22:02","guid":{"rendered":"http:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/?page_id=261"},"modified":"2017-05-12T19:54:34","modified_gmt":"2017-05-12T19:54:34","slug":"system-design-2","status":"publish","type":"page","link":"https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/system-design\/system-design-2\/","title":{"rendered":"Subsystem Design and Description"},"content":{"rendered":"<p class=\"c21 c35\"><span class=\"c9\">The system can be broadly classified into four subsystems namely: <\/span><\/p>\n<p class=\"c21 c35\"><span class=\"c9\">(1) Autonomous Flight Subsystem<\/span><\/p>\n<p class=\"c21 c35\"><span class=\"c9\">(2) Sensing Subsystem<\/span><\/p>\n<p class=\"c21 c35\">(3) Signature Detection and Analysis Subsystem<\/p>\n<p class=\"c21 c35\">(4)\u00a0Rescue Package Drop Subsystem<\/p>\n<p class=\"c21 c35\"><span class=\"c9\">Each of these subsystems is further\u00a0detailed as below:<\/span><\/p>\n<h2 id=\"h.y73dbx48a9ii\" class=\"c21 c43\"><span class=\"c30\">1. Autonomous Flight Subsystem<\/span><\/h2>\n<p class=\"c21 c35\"><span class=\"c9\">Two of the key requirements for SAR (Search and Rescue) missions are:<\/span><\/p>\n<ul class=\"c4 lst-kix_xdwmkdcbotln-0 start\">\n<li class=\"c38\"><span class=\"c9\">To be able to carry out precise waypoint navigation in a large, partially unknown environment, within strict time constraints. <\/span><\/li>\n<li class=\"c38\"><span class=\"c9\">To be able to mount a variety of powerful sensors to detect and capture possible human signatures that could be used to pinpoint the exact location for a rescue operation.<\/span><\/li>\n<\/ul>\n<p class=\"c21 c35\"><span class=\"c9\">The DJI Matrice 100\u00a0quadcopter\u00a0shown below in Figure 1 demonstrates capabilities that satisfy the above constraints and presents a very strong case for use in Aerial Search and Rescue Applications.<\/span><\/p>\n<p class=\"c112 c35\"><img decoding=\"async\" class=\"aligncenter\" title=\"\" src=\"http:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2016\/11\/Screen-Shot-2016-11-18-at-12.47.24-PM.png\" alt=\"Screen Shot 2016-09-29 at 10.09.48 AM.png\" \/><\/p>\n<p class=\"c46\" style=\"text-align: center\"><span class=\"c1\">Figure 1 DJI Matrice 100<\/span><\/p>\n<p class=\"c21 c35\"><span class=\"c9\">To be able to capture sensor data with sufficient coverage and high resolution, the system needs to establish a systematic search pattern around the area. To achieve this, the system shall implement a sweep of the provided area so that maximum coverage of the area is achieved. The pattern is illustrated in Figure 2.<\/span><\/p>\n<p class=\"c112\"><img loading=\"lazy\" decoding=\"async\" class=\"size-medium wp-image-934 aligncenter\" src=\"http:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2016\/11\/Screen-Shot-2017-04-04-at-10.37.34-PM-300x263.png\" alt=\"\" width=\"300\" height=\"263\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2016\/11\/Screen-Shot-2017-04-04-at-10.37.34-PM-300x263.png 300w, https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2016\/11\/Screen-Shot-2017-04-04-at-10.37.34-PM-768x674.png 768w, https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2016\/11\/Screen-Shot-2017-04-04-at-10.37.34-PM-1024x899.png 1024w, https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2016\/11\/Screen-Shot-2017-04-04-at-10.37.34-PM-830x729.png 830w, https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2016\/11\/Screen-Shot-2017-04-04-at-10.37.34-PM-230x202.png 230w, https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2016\/11\/Screen-Shot-2017-04-04-at-10.37.34-PM-350x307.png 350w, https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2016\/11\/Screen-Shot-2017-04-04-at-10.37.34-PM-480x421.png 480w, https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2016\/11\/Screen-Shot-2017-04-04-at-10.37.34-PM.png 1098w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/p>\n<p class=\"c46\" style=\"text-align: center\"><span class=\"c1\">Figure 2: Localized sweep\u00a0for maximizing coverage<\/span><\/p>\n<h2 id=\"h.y73dbx48a9ii\" class=\"c21 c43\"><span class=\"c30\">2. Sensing Subsystem<\/span><\/h2>\n<p class=\"c21 c35\"><span class=\"c9\">There sensing subsystem will comprise of three logical sensors namely an RGB camera, thermal camera and a \u00a0microphone for detecting voice activity.\u00a0<\/span><span class=\"c9\"> The RGB and Thermal imaging sensing capabilities are provided by a single Flir Duo sensor shown in Figure 4. Since this sensor lacks in built GPS, \u00a0alternate timestamp based techniques will be used to generate\u00a0a precise spatiotemporal overlay for the sensor data. This is required for being able to pinpoint the rescue location with high accuracy. Microphones will also be used for more precise signature detection and for breaking ties when there are multiple candidate rescue locations.\u00a0<\/span><\/p>\n<p class=\"c21 c35\"><img loading=\"lazy\" decoding=\"async\" class=\"size-medium wp-image-815 aligncenter\" src=\"http:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2017\/02\/Screen-Shot-2017-02-25-at-11.28.00-AM-300x193.png\" alt=\"\" width=\"300\" height=\"193\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2017\/02\/Screen-Shot-2017-02-25-at-11.28.00-AM-300x193.png 300w, https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2017\/02\/Screen-Shot-2017-02-25-at-11.28.00-AM-768x494.png 768w, https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2017\/02\/Screen-Shot-2017-02-25-at-11.28.00-AM-1024x658.png 1024w, https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2017\/02\/Screen-Shot-2017-02-25-at-11.28.00-AM.png 1322w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/p>\n<p class=\"c21 c35\" style=\"text-align: center\">Figure 5:\u00a0FLIR Duo Camera<\/p>\n<p class=\"c21 c43\">Microphone will also be used for more precise signature detection, which might act as an important differentiating factor when there are multiple visual cues. For our particular scenario, the sound system needs to satisfy a few criteria. First and foremost, the microphone needs to be a super cardioid microphone (depicted in Figure 6) that picks up sound only from the front of the microphone and eliminates noise from the side and the back. This is essential to keep the propeller noise at a minimum level in the collected sound samples. The next requirement is to ensure the device was capable of storing<\/p>\n<p class=\"c21 c43\">First and foremost, the microphone needs to be a super cardioid microphone (depicted in Figure 6) that picks up sound only from the front of the microphone and eliminates noise from the side and the back. This is essential to keep the propeller noise at a minimum level in the collected sound samples. The next requirement is to ensure that the device is capable of storing digitally recorded samples of sound rather than having to connect the microphone to a processing device. This requirement stems from the fact that, the microphone might have to be suspended 10-15 feet from the drone, and building a system with long data cables connecting it to a processing system on the drone would make it complicated. The third requirement is to make sure that the whole suspended system does not weigh more than a few hundred grams. This is to stay within the limits of the weight capacity for Matrice 100 which is 1kg. Lastly, it had to be something that was not very expensive so as to ensure that there is still room in the budget for buying other things.<\/p>\n<p class=\"c21 c43\">After doing a lot of research it was finally decided to go with a shotgun microphone with a built-in voice recorder that converts and stores the sound as .wav files which can be easily extracted and processed. Figure 1 below shows an image of the microphone that we decided to purchase. It has a built-in recorder that stores the data in an SD card that can be plugged into a laptop to extract the sound samples.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"size-medium wp-image-530 aligncenter\" src=\"http:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2016\/11\/Microphone-300x187.jpg\" alt=\"\" width=\"300\" height=\"187\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2016\/11\/Microphone-300x187.jpg 300w, https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2016\/11\/Microphone-250x156.jpg 250w, https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2016\/11\/Microphone-100x62.jpg 100w, https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2016\/11\/Microphone.jpg 326w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/p>\n<p class=\"c21 c43\" style=\"text-align: center\">Figure 6:\u00a0Supercardioid Microphone<\/p>\n<h2 id=\"h.y73dbx48a9ii\" class=\"c21 c43\"><span class=\"c30\">3. Signature Detection and Analysis Subsystem<\/span><\/h2>\n<p class=\"c21 c35\"><span class=\"c9\">\u00a0 The ability to detect human signatures accurately is one of the key pieces of a Search and Rescue mission. Apart from impacting design choices in other subsystems like sensors etc, the kind of signatures that are used for human detection, directly impacts the mission complexity that the system can undertake. Some of the key signatures the system will rely on are listed below:<\/span><\/p>\n<ul class=\"c4 lst-kix_1io8jtvtlz4t-0 start\">\n<li class=\"c38\"><span class=\"c9\">RGB Imagery: RGB imagery will be used to identify specific patterns of colors associated with human presence. For example, a bright colored tent in the background of a green foliage or brown terrain can be used as a strong indication of human presence in the region.<\/span><\/li>\n<li class=\"c38\"><span class=\"c9\">Thermal imagery: Due to the fact that all bodies in nature exhibit unique thermal signatures in various situations, using thermal imagery to detect the\u00a0presence of humans or objects associated with humans (a hot abandoned vehicle for example or hot wood used for fire) can yield very precise results.<\/span><\/li>\n<li class=\"c38\"><span class=\"c9\">Sound: Sound is yet another useful signature that could be associated with humans in a very distinct way. Similar to thermal fingerprint, there is a very precise sound frequency range that humans produce. This, coupled with other intuitions (associating increased amplitude with a cry for help for example) can potentially be a powerful tool in guiding a search. The one problem with using sound is to be able to filter out the surrounding noise and extract high-quality data which needs to be addressed.<\/span><\/li>\n<\/ul>\n<p class=\"c21 c35\"><span class=\"c9\">\u00a0 The system shall process the sensor data to generate a sanitized version of it. The sanitized data shall then be discretized into a set of candidates keyed by their spatial and temporal coordinates. The system will then employ advanced machine learning algorithms to classify the candidates based on the presence of signatures and generate a filtered list that is ranked by a score indicating the probability of the candidate being a rescue location. The top element in the ranked list shall be deemed as the likely rescue location to send the drone.<\/span><\/p>\n<p><span style=\"font-weight: 400\">\u00a0 With the objective of search and rescue in wilderness in mind, we decided to go after the following two types of human signatures:<\/span><span style=\"font-weight: 400\"><br \/>\n<\/span><span style=\"font-weight: 400\">1. Humans themselves<\/span><span style=\"font-weight: 400\"><br \/>\n<\/span><span style=\"font-weight: 400\">2. Signatures related to human activity: <\/span><\/p>\n<p><span style=\"font-weight: 400\">(a) Bright objects: include bright clothing, tents, or mattresses generally used while hiking,<\/span><\/p>\n<p><span style=\"font-weight: 400\">(b) Hot objects: hot stove, fire, hot water, etc which might indicate human activity<\/span><\/p>\n<p class=\"c21 c35\"><b>Human detection:<\/b><\/p>\n<p><span style=\"font-weight: 400\">Detecting humans in images is a challenging task owing to their variable appearances and wide range of poses. Our motivation behind developing an algorithm to detect the presence of human beings is that it can be used in various scenarios. More specifically, it can be applied in autonomous search and rescue operations through aerial platforms, which can effectively reduce the equipment cost and risks of injuries of humans. <\/span><\/p>\n<p class=\"c21 c35\"><span style=\"font-weight: 400\">In this project, we firstly implemented <\/span><b>Edge detection<\/b><span style=\"font-weight: 400\"> in images for capturing potential human candidates (ROIs). Then we utilized <\/span><b>HOG<\/b><span style=\"font-weight: 400\"> to extract features and classify whether there are human beings inside the ROIs based on linear support vector machine<\/span><b>(SVM)<\/b><span style=\"font-weight: 400\">. We apply this model to both the RGB images and Thermal images. Plus, we\u2019ve achieve fusing the two results and get a better result in human detection.<\/span><\/p>\n<p class=\"c21 c35\"><b>Detecting other signatures related to human activity:<\/b><\/p>\n<ol>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">We implemented bright object detection by converting the images to HSV space and thresholding them based on saturation and value to obtain bright features, which was followed by morphological operations to get bright objects. <\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">For detecting hot objects, we used adaptive thresholding on thermal images.<\/span><\/li>\n<\/ol>\n<p><span style=\"font-weight: 400\">In the Figure shown below, you can see the overview of signature detection and analysis subsystem. The modeling and analysis processes will be shown in details in the following part.<\/span><\/p>\n<p class=\"c21 c35\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-1068 aligncenter\" src=\"http:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2016\/10\/Signature-Overview.png\" alt=\"\" width=\"640\" height=\"390\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2016\/10\/Signature-Overview.png 1102w, https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2016\/10\/Signature-Overview-300x183.png 300w, https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2016\/10\/Signature-Overview-768x468.png 768w, https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2016\/10\/Signature-Overview-1024x624.png 1024w, https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2016\/10\/Signature-Overview-830x506.png 830w, https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2016\/10\/Signature-Overview-230x140.png 230w, https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2016\/10\/Signature-Overview-350x213.png 350w, https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2016\/10\/Signature-Overview-480x293.png 480w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/p>\n<h2 class=\"c21 c43\"><\/h2>\n<h2 id=\"h.y73dbx48a9ii\" class=\"c21 c43\"><span class=\"c30\">4. Rescue Package Drop Subsystem<\/span><\/h2>\n<p>The Rescue Package Drop Subsystem, as the name suggests, consists of the algorithm which estimates the GPS location of the identified signature and the payload which is responsible for dropping a rescue package drop at that location. It will be a single unit with a rescue package and a controller to actuate the dropping mechanism.<\/p>\n<p>&nbsp;<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-1003\" src=\"http:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2016\/10\/packagedrop-1024x287.png\" alt=\"\" width=\"600\" height=\"168\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2016\/10\/packagedrop-1024x287.png 1024w, https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2016\/10\/packagedrop-300x84.png 300w, https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2016\/10\/packagedrop-768x215.png 768w, https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2016\/10\/packagedrop-830x233.png 830w, https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2016\/10\/packagedrop-230x65.png 230w, https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2016\/10\/packagedrop-350x98.png 350w, https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2016\/10\/packagedrop-480x135.png 480w, https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-content\/uploads\/sites\/17\/2016\/10\/packagedrop.png 1394w\" sizes=\"auto, (max-width: 600px) 100vw, 600px\" \/><\/p>\n<p>As is clear from the graphic above, it has three important components:<\/p>\n<ol>\n<li>Rescue GPS location estimation algorithm<\/li>\n<li>Package Drop Mechanism<\/li>\n<li>System to enable autonomous package drop<\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"<p>The system can be broadly classified into four subsystems namely: (1) Autonomous Flight Subsystem (2) Sensing Subsystem (3) Signature Detection and Analysis [&hellip;]<\/p>\n","protected":false},"author":80,"featured_media":0,"parent":132,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-261","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-json\/wp\/v2\/pages\/261","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-json\/wp\/v2\/users\/80"}],"replies":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-json\/wp\/v2\/comments?post=261"}],"version-history":[{"count":39,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-json\/wp\/v2\/pages\/261\/revisions"}],"predecessor-version":[{"id":1069,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-json\/wp\/v2\/pages\/261\/revisions\/1069"}],"up":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-json\/wp\/v2\/pages\/132"}],"wp:attachment":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2016teamf\/wp-json\/wp\/v2\/media?parent=261"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}