Data Capture Subsystem

Data Capture

Data is a crucial part of our system as it brings the “realism” to our behavior model. We have identified multiple ways of getting realistic data and plan on using a blend of them for our system. We have also recognized the information that would be relevant from the data. We are now in the process of extracting the following information from it:

  1. Drivable area
  2. Lane markings
  3. Vehicles – Position and Pose
  4. Pedestrians – Position and Pose
  5. State of Traffic Light

Open Datasets

Open datasets are publicly available datasets that contain traffic data collected by driving in a given city. Each of the vehicles that have captured this data (ego vehicle) is equipped with multiple sensors that make it easier to extract objects of interest. In our case, we will be using pre-labeled data that will include the trajectories of the ego vehicle, the cars and pedestrians around it, and details of the map as well. Usage of multiple sensors make the labeling very accurate and hence reliable for our use. The data from open sources is now being converted into a format that can be used to train the behavior model. This video shows the data from Nuscenes in the ego perspective as well as in the input format.

Synthetic Data

The steering wheel based data capture system includes using a steering wheel controller(with the steering wheel and gas/brake pedals). This will be driven manually to control vehicles in the Carla simulator and realistic driven trajectories will be extracted from the simulator. This enables us to have very accurate information about actor and environment states, which can be directly utilized by the behavior model. A Logitech Driving Force G920 has been used as the Steering Wheel System. It has also been interfaced with Carla for manual control over simulator agents the and driving data is stored for every episode. We will log the data of the environment while an ego vehicle is being controlled through the steering wheel.

Video Capture from Traffic Intersections

The traffic camera-based data capture includes the video data recorded by placing a camera at a traffic light intersection. We are currently using a GoPro camera as it satisfies all of our requirements. We will use it to capture videos which would be further processed by our data processing systems for evaluating our behavior models. The primary goal of the system is to capture the view of the entire intersection along with all actors in the scene.

Initial prototyping of the capture subsystem has been done using virtual cameras in a Carla simulator environment. This has helped us identify the best mounting points for the cameras for an ideal video stream. Also, as planned, this has also allowed us to prototype and test the further subsystems.

We can vary the location of the traffic camera, the weather, and the density of the vehicles in the simulator. Also, one major advantage of using the simulator is that besides the video. we also get access to the ground truth trajectories. For the video, we transformed this ground truth to the camera frame and used it to train the detection models. We also used the real-world trajectories to evaluate the birds-eye view trajectories that we extract using the Data Processing subsystem.