Fall Validation Experiment
| Requirement Number | Requirement | Achieved |
|---|---|---|
| MP1 | Collect 10 km of synthetic sensor data and ground truth from the simulator | 16.7 Km |
| MP2 | Detect oncoming vehicles in simulator up to a distance of 100 m in with a mAP of 0.4. | 110 m with 0.6-0.9 mAP |
| MP3 | Detect lane markings with an accuracy of 75% within a maximum offset of 0.5 m. | 75-90% of frames within 0.5m offset |
| MP4 | Predict oncoming vehicle’s trajectory 2 second into the future with an RMSE of less than 3 m with respect to the ground truth trajectory. | RMSE of oncoming within 2m |
| MP5 | Predict ego-trajectory 3 seconds into the future with an RMSE of less than 2 meters with respect to the ground truth trajectory. | RMSE of ego-vehicle within 1.7m |
| MP6 | Perform tracking and fusion with MOTA (multi-object tracking accuracy) of 60% and MOTP (multi-object tracking precision) of 55%. | Achieved MOTA 75%, MOTP 80% |
| MP7 | Visualize detections with track IDs, occupancy grids with objects and road lanes, and future trajectories of oncoming vehicles at a minimum of 5 FPS. | Real-time visualization |
| MP8 | Perform perception and prediction in real-time at 10 FPS. | 21 FPS achieved |
| MP9 | Predict the possibility of a head-on collision (small overlap) in simulator with an accuracy of 80% also in 90% of the cases | Achieved 82% in over 90% cases |
| MP10 | Plan an optimal and feasible evasive maneuver trajectory for the ego-vehicle within 50ms. | Less than 1ms |
| MP11 | Ego-vehicle tracks the optimal trajectory with an error of less than 1m in Carla | Tracked within 0.52 meter |
| MP12 | Ego-vehicle camera detects the position of the other RC car within a tolerance of ±50cm of the ground truth position | Tracked with ±30cm |
| MP13 | Detect the oncoming RC car using sensor fusion within a position tolerance of ±30cm. | RMSE within 20cm upto a distance of 5m |
| MP14 | Detect the oncoming RC car velocity using sensor fusion within a tolerance of 15% of the ground truth RC car velocity | RMSE within 15% tolerance of ground truth |
Spring Validation Experiment
| No. | Requirement | Achieved |
|---|---|---|
| M.P.1 | Collect 10 kms of synthetic sensor data and ground truth from the simulator | 16.7 Km |
| M.P.2 | Detect oncoming vehicles in simulator up to a distance of 100 m in with a mAP of 0.4 | mAP of 0.6-0.9 |
| M.P.3 | Detect lane markings in simulator with an accuracy of 75% within an offset of 5% image width | 85-97% accuracy |
| M.P.6 | Predict ego-trajectory 3 seconds into the future with a RMSE of less than 2 meters w.r.t. the GT trajectory | 1.1 - 2.2 m |
| M.P.7 | Display 2D visualization of objects, detections, and trajectories at a minimum of 3 FPS | 28 FPS |
| M.P.13 | Perform perception and prediction in real-time at 10 FPS | 23-25 FPS |
| M.N.1 | Collect data from the sensor rig and visualize in Rviz (30 FPS for Point Grey) with a playback drop rate of maximum 10% | Average of 30.125 FPS with 0% drop rate |