{"id":52,"date":"2020-01-20T17:12:14","date_gmt":"2020-01-20T22:12:14","guid":{"rendered":"http:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/?page_id=52"},"modified":"2020-12-18T20:12:47","modified_gmt":"2020-12-19T01:12:47","slug":"system-performance","status":"publish","type":"page","link":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/system-performance\/","title":{"rendered":"System Performance"},"content":{"rendered":"
\n

The performance of each subsystem as a combination of the Spring and Fall Validation demonstration is discussed as follows:<\/p>\n

Data Capture<\/h1>\n

We captured videos with varied traffic densities, duration, traffic behaviors and at different intersections within CARLA. Our pipeline allows us to have explicit control over all of these parameters. We simultaneously recorded the ground truth states within the simulator, which were used for evaluation of Data Preprocessing subsystem.<\/p>\n

We also captured real-world video at a traffic intersection. This data was also used in subsequent steps to extract behavioral parameters.<\/p>\n

Simultaneous data capture from opposite sides enable 100% view of the intersection. Both CARLA camera and GoPro allow us to adjust frames per second to very high values. In our work we captured data at 30 and 60 fps respectively, hence we fulfill our mandatory requirements.<\/p>\n

Data Preprocessing<\/h1>\n

Detection<\/h3>\n

Faster RCNN Model (pre-trained on COCO and finetuned on custom dataset)<\/p>\n

<\/p>\n\n\n\n\n<\/colgroup>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ConfThreshold<\/strong>
@0.5IoU<\/strong><\/th>\n		Precision<\/strong><\/th>\nRecall<\/strong><\/th>\n<\/tr>\n
10.01%<\/td>\n72.26%<\/td>\n95.80%<\/td>\n<\/tr>\n
20.00%<\/td>\n81.71%<\/td>\n95.24%<\/td>\n<\/tr>\n
30.04%<\/td>\n86.38%<\/td>\n94.55%<\/td>\n<\/tr>\n
40.06%<\/td>\n89.92%<\/td>\n94.06%<\/td>\n<\/tr>\n
50.13%<\/td>\n92.74%<\/td>\n93.34%<\/td>\n<\/tr>\n
60.02%<\/td>\n94.38%<\/td>\n92.65%<\/td>\n<\/tr>\n
70.00%<\/td>\n95.98%<\/td>\n91.66%<\/td>\n<\/tr>\n
80.08%<\/strong><\/td>\n97.54%<\/strong><\/td>\n90.32%<\/strong><\/td>\n<\/tr>\n
90.06%<\/td>\n98.70%<\/td>\n88.18%<\/td>\n<\/tr>\n
95.05%<\/td>\n99.25%<\/td>\n85.50%<\/td>\n<\/tr>\n
98.01%<\/td>\n99.59%<\/td>\n81.21%<\/td>\n<\/tr>\n
99.00%<\/td>\n99.88%<\/td>\n76.09%<\/td>\n<\/tr>\n
99.90%<\/td>\n100.00%<\/td>\n37.74%<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

At 80% confidence threshold, we achieve both our mandatory and desired performance (Precision and Recall) for the detection model.<\/p>\n

Tracking<\/h3>\n

MOT metrics for tracking(combined result of Image and Birds Eye View tracking) calculated in the Birds Eye View space.<\/p>\n

<\/p>\n\n\n\n\n\n<\/colgroup>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Traffic Light ID<\/th>\nMOTA<\/th>\nMOTP<\/th>\nTraffic Density<\/th>\n<\/tr>\n
54<\/td>\n75.59<\/td>\n73.32<\/td>\nLow<\/td>\n<\/tr>\n
55<\/td>\n86.43<\/td>\n54.11<\/td>\nLow<\/td>\n<\/tr>\n
61<\/td>\n98.17<\/td>\n80.38<\/td>\nLow<\/td>\n<\/tr>\n
62<\/td>\n98.44<\/td>\n62.41<\/td>\nLow<\/td>\n<\/tr>\n
65<\/td>\n72.79<\/td>\n60.26<\/td>\nLow<\/td>\n<\/tr>\n
65<\/td>\n52.7<\/td>\n51.08<\/td>\nModerate<\/td>\n<\/tr>\n
62<\/td>\n89.02<\/td>\n61.19<\/td>\nModerate<\/td>\n<\/tr>\n
61<\/td>\n94.38<\/td>\n76.05<\/td>\nModerate<\/td>\n<\/tr>\n
55<\/td>\n78.85<\/td>\n76.05<\/td>\nModerate<\/td>\n<\/tr>\n
54<\/td>\n60.94<\/td>\n72.59<\/td>\nModerate<\/td>\n<\/tr>\n
65<\/td>\n65.78<\/td>\n61.66<\/td>\nDense<\/td>\n<\/tr>\n
55<\/td>\n91.19<\/td>\n80.69<\/td>\nDense<\/td>\n<\/tr>\n
Average<\/td>\n80.36<\/td>\n67.48<\/td>\n <\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Parameter Extraction<\/h3>\n\n\n\n\n\n\n\n\n\n\n
Intersection<\/th>\nTime(s)<\/th>\nNo. of Vehicles<\/th>\nGround Truth(Simulation State)<\/th>\nExtracted Params(Perception)<\/th>\n<\/tr>\n
Traffic Light Breaking Percentage<\/td>\nMin. Lead Vehicle Distance (m)<\/td>\nTraffic Light Breaking Percentage<\/td>\nMin. Lead Vehicle Distance<\/td>\n<\/tr>\n<\/thead>\n
<\/td>\n <\/td>\n <\/td>\n <\/td>\n <\/td>\nPredicted<\/td>\n% Error<\/td>\nPredicted<\/td>\n% Error<\/td>\n<\/tr>\n
1<\/td>\n30<\/td>\n125<\/td>\n90.91%<\/td>\n8.8<\/td>\n72.99%<\/td>\n19.7%<\/td>\n10.05<\/td>\n14.2%<\/td>\n<\/tr>\n
1<\/td>\n300<\/td>\n75<\/td>\n42.86%<\/td>\n7.0<\/td>\n37.5%<\/td>\n12.5%<\/td>\n5.78<\/td>\n17.42%<\/td>\n<\/tr>\n
2<\/td>\n30<\/td>\n100<\/td>\n100%<\/td>\n14.19<\/td>\n85.7%<\/td>\n14.3%<\/td>\n14.54<\/td>\n2.4%<\/td>\n<\/tr>\n
2<\/td>\n300<\/td>\n75<\/td>\n69.81%<\/td>\n8.7<\/td>\n67.5%<\/td>\n3.3%<\/td>\n8.32<\/td>\n4.3%<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

 <\/h1>\n

Modeling<\/h1>\n

Rule-based Modeling<\/h3>\n

For evaluating the performance of our rule-based modeling, we tested a wide variety of scenarios. The evaluator was allowed to choose the value of the tunable parameters to observe certain traffic scenario, which was then verified for its accuracy by observing the ground truth states from the simulation. Following were the results of the two runs during the Fall Validation Demonstration from which we can see that our percentage change from the configured values is always below 20%. Hence we meet our requirement M.P.7.<\/p>\n\n\n\n\n\n\n\n\n\n\n
Aggressive Town<\/th>\n<\/tr>\n<\/thead>\n
 <\/td>\nTraffic Light Violation(TLV) (%)<\/td>\nMin. Lead Vehicle Distance(LVD) (m)<\/td>\n<\/tr>\n
Chosen Parameters in range<\/td>\n90<\/td>\n5<\/td>\n<\/tr>\n
Observed Parameters<\/td>\n87<\/td>\n4.3<\/td>\n<\/tr>\n
Percentage Change<\/td>\n3.3<\/td>\n14<\/td>\n<\/tr>\n
Allowed Tolerance<\/td>\n20%<\/td>\n20%<\/td>\n<\/tr>\n
*Number of Vehicles: 70, *Number of Pedestrians: 20<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n\n\n\n\n\n\n\n
Moderate Town*<\/th>\n<\/tr>\n<\/thead>\n
 <\/td>\nTraffic Light Violation(TLV) (%)<\/td>\nMin. Lead Vehicle Distance(LVD) (m)<\/td>\n<\/tr>\n
Chosen Parameters in range<\/td>\n60<\/td>\n5<\/td>\n<\/tr>\n
Observed Parameters<\/td>\n48.42<\/td>\n6.5<\/td>\n<\/tr>\n
Percentage Change<\/td>\n19.3<\/td>\n0**<\/td>\n<\/tr>\n
Allowed Tolerance<\/td>\n20%<\/td>\n20%<\/td>\n<\/tr>\n
*Number of Vehicles: 70, *Number of Pedestrians: 20<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

A visualization tool was created to verify the observed parameters in real-time, it showcases the frames per second, number of vehicles, and the values of the observed parameters. The vehicles jumping traffic lights are highlighted with a red box and whenever a new leader-follower pair with minimum distance is observed it is marked with a blue arrow as shown below.<\/p>\n

\"\"<\/p>\n

Learning-based Modelling<\/h3>\n

For evaluating the improvements to the model’s efficiency, we benchmarked its size and inference speed against the previous version.
We moved from a DeepLabV3 with Resnet model to a DeepLabV3+ with Mobilenet model. This reduced the model size from ~470MB to ~60MB. This allows us to essentially spawn around 130<\/strong> agents on an 8GB GPU. Also the inference speed on our hardware went from around 500ms to around 20ms.
We also tested the maximum concurrency in our simulation environment and the compared it with the previous version. You can find more details on this in the next section.<\/p>\n

Traffic Simulation<\/h1>\n

<\/p>\n

Rule-Based Modelling<\/strong><\/h3>\n

For the rule-based modeling, we evaluated our performance using the traffic manager module of CARLA. We were able to achieve around 20 FPS with 100 simultaneous agents.<\/p>\n

Learning-Based Modelling<\/h3>\n

We have evaluated the performance of the Traffic Simulationsub-system with learning-based models by running multiple agents concurrently. We can see the table below which shows the number of customs agents vs the FPS performance of the system. As we can see that we are satisfying the mandatory performance requirement: M.P.6 (Have a simulation speed of minimum 10 FPS supporting at least 3 simultaneous agents).<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Number Of Vehicles<\/th>\nServer FPS<\/th>\n<\/tr>\n<\/thead>\n
1<\/td>\n40.0<\/td>\n<\/tr>\n
2<\/td>\n18.61<\/td>\n<\/tr>\n
3<\/strong><\/span><\/td>\n13.97<\/strong><\/span><\/td>\n<\/tr>\n
4<\/td>\n12.14<\/td>\n<\/tr>\n
5<\/td>\n9.93<\/td>\n<\/tr>\n
6<\/td>\n8.21<\/td>\n<\/tr>\n
7<\/td>\n5.68<\/td>\n<\/tr>\n
8<\/td>\n5.5<\/td>\n<\/tr>\n
9<\/td>\n5.21<\/td>\n<\/tr>\n
10<\/td>\n5.04<\/td>\n<\/tr>\n
15<\/td>\n3.6<\/td>\n<\/tr>\n
20<\/td>\n2.73<\/td>\n<\/tr>\n
30<\/td>\n1.82<\/td>\n<\/tr>\n
50<\/td>\n1.14<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

\"\"<\/p>\n\n\n

<\/p>\n","protected":false},"excerpt":{"rendered":"

The performance of each subsystem as a combination of the Spring and Fall Validation demonstration is discussed as follows: Data Capture We captured videos with varied traffic densities, duration, traffic behaviors and at different intersections within CARLA. Our pipeline allows us to have explicit control over all of these parameters. We simultaneously recorded the ground
+ Read More<\/a><\/p>\n","protected":false},"author":215,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-52","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-json\/wp\/v2\/pages\/52","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-json\/wp\/v2\/users\/215"}],"replies":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-json\/wp\/v2\/comments?post=52"}],"version-history":[{"count":34,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-json\/wp\/v2\/pages\/52\/revisions"}],"predecessor-version":[{"id":758,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-json\/wp\/v2\/pages\/52\/revisions\/758"}],"wp:attachment":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-json\/wp\/v2\/media?parent=52"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}