Test Plan

February 19, 2020 karcotsr
Off
Project Management

Spring Validation Demo

Spring Validation Demo-1

Objective

To show that the system can perform successful lane changes in straight roads. 

Validation Criteria

  • The vehicle’s speed remains in the range of 10-20mph throughout the whole manoeuvre. (PFR.3, FR.3)
  • The system can perform collision-free lane changes 80% of the time. (PFR.4, FR.4)
  • The vehicle will remain at least 0.3m away from lane markings and other actors at all times. (PFR.1, PFR.2, FR.2)

Experiment Procedure

  • A Carla scenario with a straight road containing several lanes is created.
  • Several actors, moving in a straight line on different lanes and at varying speeds in the range of 10-20 mph, are spawned.
  • An ego vehicle in the adjacent lane is spawned which is moving at 20mph.
  • The ego vehicle shall perform a lane change when a proper opportunity presents.

Spring Validation Demo-2

Objective

To show that the system can detect and stop when pedestrians are encountered

Validation Criteria

  • Ego vehicle will stop at least 0.3 metres away from the pedestrian if the pedestrian is at least 14 metres away. (PFR.2, FR.2, FR.5)
  • The ego vehicle can detect pedestrians and safely come to a halt 80% of the time. (PFR.5, FR.5)

Experiment Procedure

  • A Carla scenario of a straight lane is created and a pedestrian is spawned at a random position, at least 14 metres ahead of the ego vehicle.
  • An ego vehicle with an initial speed of 20mph is spawned.
  • The ego vehicle detects the pedestrian and decelerates to a stop.

Fall Validation Demo

Fall Validation Demo

ObjectiveThis test is designed to verify the full system performance ensuring the accumulation of errors from each sub-system satisfies the full system requirement. Additionally, it tests for the proper integration between all the sub-systems.
PersonnelFull Team
EquipmentA Computer with
1. Appropriate version of CARLA installed.
2. CARLA testing maps stored.
3. ROS installed.
A mouse with a scroll. (For visualization view point movements)
LocationRemote
ElementsUtility Classes, Simulation, Trajectory Generator, Reinforcement Learning Agents, Neural Network Selector, State Manager
ProcedureThis test needs to be conducted for different combination of scenarios, the procedure for testing each combination is as follows:
1. Start the CARLA Simulator.
2. Launch the Path Planner Node and CARLA Interface Node.
3. Verify the connection between all ROS nodes and CARLA simulator.
4. Initialize the test case with a pre-defined parameter file that specifies: map name, starting pose, destination pose, initial state of other actors, and max allowed time.
5. The controlled vehicle is expected to travel safely from the starting pose to the destination pose within the max allowed time.

The procedure above will be repeated for the following cases:
1. A case that involves at least 1 lane change, 1 left-turn, and 1 straight intersection.
2. A case that involves at least 1 lane change and 1 right-turn.
3. A case that involves pedestrian jay-walking.
4. A case that involves at least 1 left-turn that has pedestrians crossing the road.
5. A case that involves at least 1 right-turn that has pedestrians crossing the road.
Verification CriteriaWe require an overall success rate of at least 70%, and a successful result is evaluated by all of the following criteria:
1.The vehicle stops within 2 meters of the goal pose and signals a job done.
2.Goal pose is reached within the max allowed time.
3.The vehicle does not collide with any other object. (including pedestrians, other vehicles, and any static object)
4.The vehicle only takes a left turn in the left-most lane.
5.The vehicle only arrives at the left-most lane after a left turn.
6.The vehicle only takes a right turn in the right-most lane.
7.The vehicle only arrives at the right-most lane after a right turn.
8.While proceeding straight in an intersection, the vehicle only crosses the stopline in green light, yellow light, or within the first 0.3 seconds of the red light.
9.While taking left-turn in an intersection, the vehicle only crosses the stopline in green light, yellow light, or within the first 0.3 seconds of the red light.
10.The vehicle does not exceed the speed limit by more than 10%.

For a detailed test plan please checkout this report.

Project Management
Project Management