System Performance

We targeted these requirements during the Spring Semester for the Spring Validation Demo(SVD),

1.Data Capture

  • M.P.1 Capture video stream at a speed of at least 30 FPS
  • M.P.2 Capture 100% view of intersection

2.Data Processing

  • M.P.3 Have detection recall of more than 75%
  • M.P.4 Have detection precision of more than 75%
  • M.P.5 MOTA (multi-object tracking accuracy) should be at least 40% and MOTP(multi-object tracking precision) should be at least 40%
  • D.P.1 Have detection recall of 90% or more
  • D.P.2 Have detection precision of 90% or more

3.Traffic Simulation

  • M.P.6 Have a simulation speed of minimum 10 FPS supporting at least 3 simultaneous agents

Our goal was to achieve the major requirements of 3 out of 4 subsystems. This gives us a good amount of time for Behavior Modelling, which will be primarily targeted during the Fall semester.


The performance of each subsystem is as follows

Detection

Faster RCNN Model (pre-trained on COCO and finetuned on custom dataset)

 

ConfThreshold
@0.5IoU
Precision Recall
10.01% 72.26% 95.80%
20.00% 81.71% 95.24%
30.04% 86.38% 94.55%
40.06% 89.92% 94.06%
50.13% 92.74% 93.34%
60.02% 94.38% 92.65%
70.00% 95.98% 91.66%
80.08% 97.54% 90.32%
90.06% 98.70% 88.18%
95.05% 99.25% 85.50%
98.01% 99.59% 81.21%
99.00% 99.88% 76.09%
99.90% 100.00% 37.74%

At 80% confidence threshold, we achieve both our mandatory and desired performance (Precision and Recall) for the detection model.

Tracking

MOT metrics for tracking(combined result of Image and Birds Eye View tracking) calculated in the Birds Eye View space.

 

Traffic Light ID MOTA MOTP Traffic Density
54 75.59 73.32 Low
55 86.43 54.11 Low
61 98.17 80.38 Low
62 98.44 62.41 Low
65 72.79 60.26 Low
65 52.7 51.08 Moderate
62 89.02 61.19 Moderate
61 94.38 76.05 Moderate
55 78.85 76.05 Moderate
54 60.94 72.59 Moderate
65 65.78 61.66 Dense
55 91.19 80.69 Dense
Average 80.36 67.48

 

Simulator Performance with Baseline behavioral model for vehicles

Number of Vehicles FPS
0 27.28
1 16.66
2 11.32
3 10.61
4 8.43
5 6.38
6 5.66
7 4.92
8 4.36
9 3.82
10 3.64