System Implementation

Infrastructure:

  • Creating a miniature setup to mimic scenario of a parking lot
  • The setup allows testing forward, backward, and parallel parking scenarios

PERCIV mostly relies on external sensing for teleoperation of RC cars in the miniaturized parking lot scenario. Currently, the external sensing involves using RGB cameras which will be expanded to incorporate depth sensing to output point clouds of the environment. Some data from onboard sensors are also used. This involves reading odometry data for estimating the position of the car and proximity sensors for estimating nearby obstacles. The onboard data is transmitted wirelessly to a computer through a ROS2 DDS node communication. External sensors are connected to the local network over a wired connection and transmitted to a control module (computer).

Figure 1. PERCIV Environment setup

Hardware Setup:

  • ROSMASTER R2 is a mobile car with Ackerman steering structure developed based on ROS system.
  • Ultrasonic sensors are mounted for collision detection and safety pipeline.

Figure 2. PERCIV Jetacker Car

Perception Subsystem:

Currently the perception pipeline consists of the following flow:

  1. Receive time-synchronized image frames from the intel Realsense using software implementation of GenLock, in which a trigger is sent programmatically to the realsense sensors to control the shutters and exposures.
  2. Once the time-synchronized images are received, they are stitched using a Graph-Neural Network based stitcher optimized to run in realtime and to operate in feature sparse environments.
  3. Once the scene is stitched, points are selected manually to perform perspective transform into a BEV. Inverse Perspective Mapping was also experimented, but due to better fidelity of a simple homography using perspective transformation in our case we decided to stick with perspective transforms. The final stitched BEV can be seen in Figure 3.

Figure 3. Stitched BEV output

  1. With the BEV generated a learning-based algorithm is used for keypoint detection for explicitly pose estimation of the ego-vehicle. The mAP (mean average precision) we were getting against the ground-truth (measure using an ArUCO marker) was 97.5%. The estimated pose can be seen in Figure 4:

Figure 4: Estimated pose estimation for the ego-vehicle

  1. With the pose of the ego-vehicle determined the kinodynamic model is used complemented by the user input to generate trajectories as a t+nth timestep based on the ackermann steering. The predicted trajectories in green can be seen in Figure 5 below:

Figure 5: Predicted trajectories using estimated pose and controller input

  1. Finally an independent greedy-heuristic for box-to-box center alignment and pose alignment to detect successful parking. Figure 6 depicts the translucent green overlay achieved as a result of successful parking:

Figure 6: Green overlay depicting successful parking and within the designated tolerance

Figure 7: Pose estimation of the car in the PERCIV environment

Controls Subsystem:

  • Steering and Throttle Input: For our setup, we used the Logitech G29 steering and pedal setup. We fine-tuned the force feedback steering option to replicate the sensation of maneuvering a real vehicle, ensuring an immersive driving experience. The pedal shifters integrated into the steering wheel offer gear changes, allowing drivers to transition through three forward gears and one reverse gear, which is also displayed on the UI/UX. The plus-minus button on the wheel can be used to change the length of the trajectory prediction. The red button can be used to disable the safety checking vehicle driving the car.        
  • Trajectory prediction: Trajectory prediction stands as a pivotal feature, offering drivers a proactive understanding of the vehicle’s forthcoming position based on present throttle and steering inputs. This tool enhances situational awareness, allowing drivers with foresight to navigate with precision and confidence. Because we are using an Ackerman type of car, we used the peppy model to predict car position and orientation given the steering commands. The lookahead distance was based on the length that the driver selected using the plus-minus button on the steering.
  • Collision Avoidance: In order to mimic the actual car, we had 3 sensors in the front and 1 sensor in the back, as shown in figure 8. These were low-cost proximity sensors and a very narrow FOV of 5 degrees. The logic involved reading these ultrasound sensor data topics, as well as the teleoperation setup input data topic (raw_cmd_vel) to filter out unsafe command velocity components and only publish velocities that are safe to execute (do not move the car in the direction of possible collisions). Therefore, these safe velocities are published in a ROS topic (cmd_vel). These safe velocities are then subscribed directly to the car and executed.
  • Offering a range of functionalities to control the car includes managing gear shifts, steering, and pedals, as well as processing user input to navigate the vehicle within the environment.
  • Future works includes incorporating cruise control and navigation using waypoints.

Figure 8: Sensor placement on the car

Figure 9: Demonstration of Trajectory prediction based on steering configuration

UI/UX:

Inherently, the complete system is a human-machine interface with UI/UX as a core submodule of the complete system. Various off-the-shelf solutions which included ROS web-bridge integration with bootstrap, Glovo, and simple HTML CSS based designs, but due to limited flexibility to change the UI/UX and high latency/low throughput from the web-bridge we opted to design our custom UI/UX design using StreamLit and ReactJS. The communication between the frontend and backend was achieved using ROS topics with high compression rates for image frames for high throughput. The elements included a side navigation bar for selecting between raw CCTV footage, pre-defined parameters for stitched BEV, and user-defined parameters for stitched BEV. In addition, the user was provided with a real time feed of the environment bolstered by other graphical interfaces which included a speedometer, gear status, and safety override status. Figure 12 shows the current status of our UI/UX:

Figure 10: UI/UX depicting stitched BEV, speedometer, gear status, and safety override status

  • Multiple Visualization Views: Offering diverse visualizations for enhanced understanding.
  • Integration of Control and Perception Data: Displaying data from both control and perception layers for comprehensive insight.
  • Bird’s Eye View (BEV) Display: Presenting a BEV of the environment for improved spatial awareness.
  • Predicted Trajectory Visualization: Displaying predicted trajectories of the car to aid in navigation.
  • Real-Time Indicators: Providing real-time indicators such as current speed, collision alerts, and corridor breach alerts for situational awareness.