{"id":283,"date":"2020-04-03T15:16:30","date_gmt":"2020-04-03T19:16:30","guid":{"rendered":"http:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/?page_id=283"},"modified":"2020-12-18T17:52:56","modified_gmt":"2020-12-18T22:52:56","slug":"preprocessing-subsystem","status":"publish","type":"page","link":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/preprocessing-subsystem\/","title":{"rendered":"Data Processing Subsystem"},"content":{"rendered":"<h1><strong>Overview<\/strong><\/h1>\n<p>The Data Processing subsystem has inputs as the data captured from data capture and outputs as the data required for our modelling subsystem.<\/p>\n<h2><strong>System Implementation Details<\/strong><\/h2>\n<p>There are two main components in our Data Processing pipeline-<\/p>\n<ul>\n<li style=\"list-style-type: none\">\n<ul>\n<li style=\"list-style-type: none\">\n<ul>\n<li>\n<h3><strong>Vehicle detection<\/strong><\/h3>\n<p>The object detection is done using Detectron2. The pretrained network on COCO dataset seems to give decent results in Carla.<br \/>\nHowever, to improve detection performance, we have extracted images and annotations from Carla. This data was captured from Carla by mounting virtual cameras at different traffic intersections. We have around 2688 training images and 672 Test images from a variety of intersections.<\/li>\n<li>\n<h3><strong>Vehicle tracking<\/strong><\/h3>\n<ul>\n<li>\n<h4><strong>SORT<\/strong><\/h4>\n<p>The 2D detected bounding boxes from the detector are tracked in the camera view. Simple and Online Realtime Tracking (SORT) is an EKF tracker tracking the 2d rectangular box&#8217;s location and size. \\\\<br \/>\nThe IDs tracked from the tracker are used further to track these locations in the bird&#8217;s eye view shown in figure.<\/p>\n<div id=\"attachment_180\" style=\"width: 610px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-180\" class=\"alignnone size-full wp-image-167\" src=\"http:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-content\/uploads\/sites\/47\/2020\/04\/realworldetection.png\" alt=\"\" width=\"600\" height=\"600\"><p id=\"caption-attachment-180\" class=\"wp-caption-text\">Object detection and tracking in real world<\/p><\/div>\n<p><div id=\"attachment_180\" style=\"width: 610px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-180\" class=\"alignnone size-full wp-image-167\" src=\"http:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-content\/uploads\/sites\/47\/2020\/04\/carladetection.png\" alt=\"\" width=\"600\" height=\"600\"><p id=\"caption-attachment-180\" class=\"wp-caption-text\">Object detection and tracking in Carla. Simple and Online Realtime Tracking(SORT) assigns id to each of the bounding boxes returned by the tracker.<\/p><\/div><\/li>\n<li>\n<h3><strong>Homography computation<\/strong><\/h3>\n<p>Once we have a list of detections for each unique object, we now want their trajectories in an absolute frame of reference. We use corresponding points in the image and the real world to calculate the homography matrix. We will transform the bottom-center point of the 2D detection box to the real world, which gives an approximate position of the object in the map. Doing this for all the detection boxes gives us a trajectory for each of the objects in the birds eye view.<\/p>\n<p>The figure below shows two views both of which are captured from Carla.<br \/>\nIn the real world we intend to capture bird eye views using google earth images as shown in figure below.<br \/>\nFrom a traffic intersection the correspondences are labelled manually for at least 4 points, and 8-point homography is computed.<\/p>\n<div id=\"attachment_544\" style=\"width: 1034px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-544\" class=\"wp-image-544 size-large\" src=\"http:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-content\/uploads\/sites\/47\/2020\/05\/carla_bothviews-1024x321.png\" alt=\"\" width=\"1024\" height=\"321\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-content\/uploads\/sites\/47\/2020\/05\/carla_bothviews-1024x321.png 1024w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-content\/uploads\/sites\/47\/2020\/05\/carla_bothviews-300x94.png 300w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-content\/uploads\/sites\/47\/2020\/05\/carla_bothviews-768x241.png 768w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-content\/uploads\/sites\/47\/2020\/05\/carla_bothviews.png 1517w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><p id=\"caption-attachment-544\" class=\"wp-caption-text\">8 point homography is computed using two views giving manual correspondences.<\/p><\/div>\n<p><div id=\"attachment_543\" style=\"width: 1034px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-543\" class=\"wp-image-543 size-large\" src=\"http:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-content\/uploads\/sites\/47\/2020\/05\/realworld_bothviews-1024x306.png\" alt=\"\" width=\"1024\" height=\"306\" srcset=\"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-content\/uploads\/sites\/47\/2020\/05\/realworld_bothviews-1024x306.png 1024w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-content\/uploads\/sites\/47\/2020\/05\/realworld_bothviews-300x90.png 300w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-content\/uploads\/sites\/47\/2020\/05\/realworld_bothviews-768x230.png 768w, https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-content\/uploads\/sites\/47\/2020\/05\/realworld_bothviews.png 1482w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><p id=\"caption-attachment-543\" class=\"wp-caption-text\">The same approach can be used in the real world dataset by using google images and traffic cam view to compute homography.<\/p><\/div><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<ul>\n<li>\n<h3><strong>HD Maps<\/strong><\/h3>\n<p>We use our custom HDMaps which allow us to know the lanes and the direction in which vehicles should go.<br \/>\nThese act as a prior information for the vehicles for BEV Tracking and allow us to identify and avoid spurious tracking predictions. A map of a real world intersection can be seen in figure below.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-184 aligncenter\" src=\"http:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-content\/uploads\/sites\/47\/2020\/12\/HD_map_resized.png\" alt=\"\" width=\"421\" height=\"215\"><\/p>\n<h3><strong>BEV Tracking<\/strong><\/h3>\n<p>The bottom centre of the 2D bounding box is transformed to BEV.<br \/>\nThe tracker\u2019s prediction uses a constant velocity motion model in the pixel space to predict the location of the vehicles.The data association was treated as a linear assignment problem and was solved using Hungarian\u2019s method. However, we do use the correct ids from SORT as priors.<\/li>\n<li>\n<h4><strong>Tracking Evaluation<\/strong><\/h4>\n<p>Final tracked pixel values are evaluated against the ground truth for both the association and deviation of predicted trajectories.<br \/>\nThe 3D location of each vehicle is transformed to bird&#8217;s eye view by the camera intrinsics and transformation given by Carla to generate ground truth pixel locations.<br \/>\nThe Multi-object metrics are used to evaluate the position predicted by tracker against ground truth locations.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-184 aligncenter\" src=\"http:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-content\/uploads\/sites\/47\/2020\/05\/bev_tracker.png\" alt=\"\" width=\"849\" height=\"422\"><\/p>\n<p>BEV Tracking. The tracker performance is computed online and is shown on the bottom right.<\/li>\n<li>\n<h3>Traffic Light State Detection<\/h3>\n<p>We trained a custom CNN to detect each of the traffic light states at an intersection, at each frame. The detected traffic light can be seen in the figure below.<br \/>\n<img loading=\"lazy\" decoding=\"async\" class=\"wp-image-184 aligncenter\" src=\"http:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-content\/uploads\/sites\/47\/2020\/12\/tl_detection.png\" alt=\"\" width=\"633\" height=\"322\"><\/li>\n<li>\n<h3>Parameter Extraction<\/h3>\n<p>All the information extracted above allows us to extract behavioral parameters which are to be replicated inside the simulator.<br \/>\nWe focus on extracting these parameters for two behaviours-<\/p>\n<ul>\n<li>Traffic light violation<\/li>\n<li>Minimum Leading Vehicle Distance<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-183 aligncenter\" src=\"http:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-content\/uploads\/sites\/47\/2020\/12\/lvd_pair.png\" alt=\"\" width=\"633\" height=\"322\"><br \/>\n<img loading=\"lazy\" decoding=\"async\" class=\"wp-image-189 aligncenter\" src=\"http:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-content\/uploads\/sites\/47\/2020\/12\/tl_violation.png\" alt=\"\" width=\"633\" height=\"322\"><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Overview The Data Processing subsystem has inputs as the data captured from data capture and outputs as the data required for our modelling subsystem. System Implementation Details There are two main components in our Data Processing pipeline- Vehicle detection The object detection is done using Detectron2. The pretrained network on COCO dataset seems to give<br \/><a class=\"moretag\" href=\"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/preprocessing-subsystem\/\">+ Read More<\/a><\/p>\n","protected":false},"author":217,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-283","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-json\/wp\/v2\/pages\/283","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-json\/wp\/v2\/users\/217"}],"replies":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-json\/wp\/v2\/comments?post=283"}],"version-history":[{"count":37,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-json\/wp\/v2\/pages\/283\/revisions"}],"predecessor-version":[{"id":653,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-json\/wp\/v2\/pages\/283\/revisions\/653"}],"wp:attachment":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-json\/wp\/v2\/media?parent=283"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}