{"id":494,"date":"2019-05-10T19:45:47","date_gmt":"2019-05-10T19:45:47","guid":{"rendered":"http:\/\/mrsdprojects.ri.cmu.edu\/2018teama\/?page_id=494"},"modified":"2020-04-11T17:49:39","modified_gmt":"2020-04-11T17:49:39","slug":"system-performance","status":"publish","type":"page","link":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teama\/system-performance\/","title":{"rendered":"Performance"},"content":{"rendered":"

Fall Validation Experiment<\/strong><\/p>\n

\n\n\n\n\n\t\n\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t
Requirement Number<\/th>Requirement<\/th>Achieved<\/th>\n<\/tr>\n<\/thead>\n
MP1<\/td>Collect 10 km of synthetic sensor data and ground truth from the simulator<\/td>16.7 Km<\/td>\n<\/tr>\n
MP2<\/td>Detect oncoming vehicles in simulator up to a distance of 100 m in with a mAP of 0.4.<\/td>110 m with 0.6-0.9 mAP<\/td>\n<\/tr>\n
MP3<\/td>Detect lane markings with an accuracy of 75% within a maximum offset of 0.5 m.<\/td>75-90% of frames within 0.5m offset<\/td>\n<\/tr>\n
MP4<\/td>Predict oncoming vehicle\u2019s trajectory 2 second into the future with an RMSE of less than 3 m with respect to the ground truth trajectory.<\/td>RMSE of oncoming within 2m<\/td>\n<\/tr>\n
MP5<\/td>Predict ego-trajectory 3 seconds into the future with an RMSE of less than 2 meters with respect to the ground truth trajectory.<\/td>RMSE of ego-vehicle within 1.7m<\/td>\n<\/tr>\n
MP6<\/td>Perform tracking and fusion with MOTA (multi-object tracking accuracy) of 60% and MOTP (multi-object tracking precision) of 55%.<\/td>Achieved MOTA 75%, MOTP 80%<\/td>\n<\/tr>\n
MP7<\/td>Visualize detections with track IDs, occupancy grids with objects and road lanes, and future trajectories of oncoming vehicles at a minimum of 5 FPS.<\/td>Real-time visualization<\/td>\n<\/tr>\n
MP8<\/td>Perform perception and prediction in real-time at 10 FPS.<\/td>21 FPS achieved<\/td>\n<\/tr>\n
MP9<\/td>Predict the possibility of a head-on collision (small overlap) in simulator with an accuracy of 80% also in 90% of the cases<\/td>Achieved 82% in over 90% cases<\/td>\n<\/tr>\n
MP10<\/td>Plan an optimal and feasible evasive maneuver trajectory for the ego-vehicle within 50ms.<\/td>Less than 1ms<\/td>\n<\/tr>\n
MP11<\/td>Ego-vehicle tracks the optimal trajectory with an error of less than 1m in Carla<\/td>Tracked within 0.52 meter<\/td>\n<\/tr>\n
MP12<\/td>Ego-vehicle camera detects the position of the other RC car within a tolerance of \u00b150cm of the ground truth position<\/td>Tracked with \u00b130cm<\/td>\n<\/tr>\n
MP13<\/td>Detect the oncoming RC car using sensor fusion within a position tolerance of \u00b130cm.<\/td>RMSE within 20cm upto a distance of 5m<\/td>\n<\/tr>\n
MP14<\/td>Detect the oncoming RC car velocity using sensor fusion within a tolerance of 15% of the ground truth RC car velocity<\/td>RMSE within 15% tolerance of ground truth<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n<\/div>\n

Spring Validation Experiment<\/strong><\/p>\n\n\n\n\n\t\n\n\t\n\t\n\t\n\t\n\t\n\t\n\t
No.<\/th>Requirement<\/th>Achieved<\/th>\n<\/tr>\n<\/thead>\n
M.P.1
\n<\/td>		Collect 10 kms of synthetic sensor data and ground truth from the simulator
\n<\/td>		16.7 Km
\n<\/td>\n<\/tr>\n
M.P.2
\n<\/td>		Detect oncoming vehicles in simulator up to a distance of 100 m in with a mAP of 0.4
\n<\/td>		mAP of 0.6-0.9
\n<\/td>\n<\/tr>\n
M.P.3
\n<\/td>		Detect lane markings in simulator with an accuracy of 75% within an offset of 5% image width
\n<\/td>		85-97% accuracy
\n<\/td>\n<\/tr>\n
M.P.6
\n<\/td>		Predict ego-trajectory 3 seconds into the future with a RMSE of less than 2 meters w.r.t. the GT trajectory
\n<\/td>		1.1 - 2.2 m
\n<\/td>\n<\/tr>\n
M.P.7
\n<\/td>		Display 2D visualization of objects, detections, and trajectories at a minimum of 3 FPS
\n<\/td>		28 FPS
\n<\/td>\n<\/tr>\n
M.P.13
\n<\/td>		Perform perception and prediction in real-time at 10 FPS
\n<\/td>		23-25 FPS
\n<\/td>\n<\/tr>\n
M.N.1
\n<\/td>		Collect data from the sensor rig and visualize in Rviz (30 FPS for Point Grey) with a playback drop rate of maximum 10%
\n<\/td>		Average of 30.125 FPS with 0% drop rate
\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n","protected":false},"excerpt":{"rendered":"

Fall Validation Experiment Spring Validation Experiment<\/p>\n","protected":false},"author":178,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-494","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teama\/wp-json\/wp\/v2\/pages\/494","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teama\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teama\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teama\/wp-json\/wp\/v2\/users\/178"}],"replies":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teama\/wp-json\/wp\/v2\/comments?post=494"}],"version-history":[{"count":60,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teama\/wp-json\/wp\/v2\/pages\/494\/revisions"}],"predecessor-version":[{"id":852,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teama\/wp-json\/wp\/v2\/pages\/494\/revisions\/852"}],"wp:attachment":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2018teama\/wp-json\/wp\/v2\/media?parent=494"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}