{"id":60,"date":"2020-01-20T17:13:58","date_gmt":"2020-01-20T22:13:58","guid":{"rendered":"http:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/?page_id=60"},"modified":"2020-10-08T14:48:28","modified_gmt":"2020-10-08T18:48:28","slug":"test-plan","status":"publish","type":"page","link":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/project-management\/test-plan\/","title":{"rendered":"Test Plan"},"content":{"rendered":"<h1>Spring Test Plan<\/h1>\n<p><b>Equipment Required<\/b><span style=\"font-weight: 400\">: Desktop system running Carla and the S.T.A.R.S. data processing pipeline, Steering Wheel System such as Logitech G920<\/span><\/p>\n<h3><b>Demonstration 1:<\/b><\/h3>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Objective:\u00a0 To demonstrate the detection algorithm by extracting the actors\u2019(vehicles) information from real-world\/Carla video streams and visualizing them.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Procedure:<\/span>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Capture visual streams form data captured from Carla\/real world.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Run our detector model on the camera stream, visualizing a bounding box showing the class of the detected actor.<\/span><\/li>\n<\/ul>\n<\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Validation Criteria:\u00a0<\/span>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Detection Precision- At least\u00a0<\/span><b>75%<\/b><span style=\"font-weight: 400\"> of the detected vehicles are actually vehicles.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Detection Recall- At least\u00a0<\/span><b>75%<\/b><span style=\"font-weight: 400\"> of the real-world objects are detected by the algorithm.\u00a0<\/span><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h3><b>Demonstration 2:<\/b><\/h3>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Objective: To demonstrate the tracking algorithm by extracting the actors\u2019(vehicles) information from video streams and visualizing them.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Procedure:<\/span>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Capture visual streams form data captured from Carla\/real world and run the detection model as in demonstration 1.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">The detection output goes through the tracking algorithm which assigns an id to each actor and tracks its location in a bird\u2019s eye view.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">The visualization shows each detected actor with its class and its id.<\/span><\/li>\n<\/ul>\n<\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Validation Criteria:<\/span>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">MOTA (multi-object tracking accuracy) should be at least\u00a0<\/span><b>40%<\/b><span style=\"font-weight: 400\"> and MOTP (multi-object tracking precision) should be at least<\/span><b>\u00a040%.\u00a0<\/b><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h3><b>Demonstration 3<\/b><\/h3>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Objective: To demonstrate the capability to control multiple modelled actors inside a simulator.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Procedure:<\/span>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">We spawn multiple actors inside Carla controlled by its default rule-based behavioral model.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">We compute the performance of Carla as we add actors controlled by our baseline behavioral model subsequently to the simulation environment.<\/span><\/li>\n<\/ul>\n<\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Validation Criteria:\u00a0<\/span>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">The simulation speed in CARLA is <\/span><b>\u226510<\/b><span style=\"font-weight: 400\"> frames per second on adding\u00a0 <\/span><b>\u22653 <\/b><span style=\"font-weight: 400\">actors (controlled by our baseline behavioral model) simultaneously.<\/span><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h1>Fall Test Plan<\/h1>\n<p><b>Equipment<span style=\"font-weight: 400\">: Desktop system running Carla and the S.T.A.R.S. pipeline, Steering wheel system such as Logitech G920, Data Capture Unit (DCU) designed and fabricated by S.T.A.R.S.<\/span><\/b><\/p>\n<h3><b>Demonstration 1:<\/b><span style=\"font-weight: 400\">\u00a0<\/span><\/h3>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Objective: To demonstrate various data capturing methods and pre-processing pipeline<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Procedure:<\/span>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">The user drives the steering wheel system in the CARLA environment running on the desktop system. The video data from the user perspective and bird\u2019s eye view is stored on the hard drive.\u00a0<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">The user mounts the DCU on a tripod at a traffic signal and captures multiple views of an intersection. The data shown will be pre-recorded.\u00a0<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Video streams from the above data sources are then fed into a laptop, detection and tracking algorithms run on it.<\/span><\/li>\n<\/ul>\n<\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Validation Criteria:\u00a0<\/span>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Data Capture Unit successfully mounted on the frame, collects data <\/span><b>\u2265 30<\/b><span style=\"font-weight: 400\"> frames per second.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Data Capture Unit enables a\u00a0<\/span><b>100%<\/b><span style=\"font-weight: 400\"> view of the intersection.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">The system detects <\/span><b>75%<\/b><span style=\"font-weight: 400\"> of actors (cars and pedestrians) seen in the input video.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">MOTA (multi-object tracking accuracy) should be <\/span><b>40%<\/b><span style=\"font-weight: 400\"> and MOTP(multi-object tracking precision) should be <\/span><b>40%.<\/b><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h3><b>Demonstration 2:<\/b><\/h3>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Objective: To demonstrate the final traffic behavior model developed by S.T.A.R.S.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Procedure:<\/span>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">The scenario from demonstration 1 is used. All actors are given their start pose and final goals. The S.T.A.R.S model will be running as the ego vehicle.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">In another sub-demonstration, 3 to 5 S.T.A.R.S. models will be running in a loop, given their goals.<\/span><\/li>\n<\/ul>\n<\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Validation Criteria:\u00a0<\/span>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Will have <\/span><b>x%<\/b><span style=\"font-weight: 400\"> mean squared error between the predicted trajectory from the behavioral model and ground truth trajectory in a given scenario.<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Carla runs at <\/span><b>\u2265 10<\/b><span style=\"font-weight: 400\"> frames per second with <\/span><b>\u2265 3<\/b><span style=\"font-weight: 400\"> actors simultaneously.<\/span><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h3><b>Demonstration 3<\/b><span style=\"font-weight: 400\">:\u00a0<\/span><\/h3>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Objective: To demonstrate that the system can be tuned for aggression.\u00a0<\/span><\/li>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Procedure:<\/span>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Given a S.T.A.R.S. various parameters will be changed as part of aggression tuning. Multiple these models will be started with such configuration.\u00a0<\/span><\/li>\n<\/ul>\n<\/li>\n<li style=\"font-weight: 400\"><i><span style=\"font-weight: 400\">Validation Criteria:<\/span><\/i><span style=\"font-weight: 400\">\u00a0<\/span>\n<ul>\n<li style=\"font-weight: 400\"><span style=\"font-weight: 400\">Allow for tuning aggression parameters of models.<\/span><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h1>End-To-End System Test Plan<\/h1>\n<p><a href=\"https:\/\/drive.google.com\/file\/d\/1dAoI_GGAlJjR5Ug4lzEaNGifhz3TXo3t\/view?usp=sharing\">Test Plan Document<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Spring Test Plan Equipment Required: Desktop system running Carla and the S.T.A.R.S. data processing pipeline, Steering Wheel System such as Logitech G920 Demonstration 1: Objective:\u00a0 To demonstrate the detection algorithm by extracting the actors\u2019(vehicles) information from real-world\/Carla video streams and visualizing them. Procedure: Capture visual streams form data captured from Carla\/real world. Run our detector<br \/><a class=\"moretag\" href=\"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/project-management\/test-plan\/\">+ Read More<\/a><\/p>\n","protected":false},"author":215,"featured_media":0,"parent":55,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-60","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-json\/wp\/v2\/pages\/60","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-json\/wp\/v2\/users\/215"}],"replies":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-json\/wp\/v2\/comments?post=60"}],"version-history":[{"count":8,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-json\/wp\/v2\/pages\/60\/revisions"}],"predecessor-version":[{"id":568,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-json\/wp\/v2\/pages\/60\/revisions\/568"}],"up":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-json\/wp\/v2\/pages\/55"}],"wp:attachment":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2020teamh\/wp-json\/wp\/v2\/media?parent=60"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}