Simulation Subsystem

Overview

This subsystem is the primary subsystem that our users and customers will use. The system will be used as testing grounds for self-driving car software and expose them to realistic scenarios. We have primarily used CARLA software as the simulation platform for our project.

The simulation subsystem will initialize the actors and their initial plans randomly inside the map. This configuration will decide the number of actors of each type, along with the actor behavior parameters and their initial starting points and destination. The configuration also contains the information if the model of the agent is rule based or learning based. After this, the model will be executed to generate realistic behaviors.

The System diagram for running a simulation

For rule based simulation, we utilize the Traffic Manager module of the CARLA simulator. This module allows us to set/change parameters of the agents like traffic breaking light percentage, ignore traffic light percentage, global minimum distance to leading vehicle and soon.

For a learning based simulation, we utilize author’s implementation of the paper Learning by Cheating as our base implementation. We use the client side Python APIs for communication with CARLA and integrate with the PyTorch library for training and model inference.

Targeted Requirements

  • M.P. 6 Have a simulation speed of minimum 10 FPS supporting at least 3 simultaneous agents.

Analysis and Testing

Traffic Simulation sub-system is primarily developed using CARLA, Traffic Manager, and Learning by Cheating. The major performance metric for this sub-system is the server frames per second (FPS). This metric is important because this will determine the speed at which the user will be able to run their agents. This will directly impact the duration of the tests the user has to run. In the figure below, we can see our test results of how server FPS drops with progressively adding more learning based agents. We are measuring server FPS continuously in the simulation, while adding more agents.

We also used profiling tools such as cProfile to understand which parts of code are taking more time for execution. This helped us understand the execution flow and optimize functions to optimize performance. Upon detailed analysis, we found out that our model execution takes about 10ms which means in theory we can provide 100 FPS. The only reason we are not able to achieve that is because the simulator CARLA takes a constant amount of time per image that is requested. In our case each model iteration requires an image of the top view from the Car’s position(Bird’s Eye view). As the number of agent’s increases, the image extraction time also increases. This makes overall fps about 13 for three agents.

Strong/Weak Points

Strengths

  • Current system performs as per the requirements. We are able to provide decent simulation speed for testing out the behavior models.
  • We have an end-to-end system working with a choice of learning-based model or the rule-based model. With this we can provide diverse edge-case scenarios.
  • Our current system is highly optimized for inference with the only bottleneck being the throughput of the CARLA simulator.

Weaknesses

  • Current implementation of CARLA provides almost no room to accommodate any new development on the server side to improve the performance (speed) of the simulation.

Results

We can see one such scenario playing out here with our baseline model.