{"id":955,"date":"2023-09-04T23:49:49","date_gmt":"2023-09-04T23:49:49","guid":{"rendered":"https:\/\/mrsdprojects.ri.cmu.edu\/2023teamf\/?page_id=955"},"modified":"2023-11-25T05:07:30","modified_gmt":"2023-11-25T05:07:30","slug":"system-implementation","status":"publish","type":"page","link":"https:\/\/mrsdprojects.ri.cmu.edu\/2023teamf\/system\/system-implementation\/","title":{"rendered":"System Implementation"},"content":{"rendered":"\n

The overall system comprises the Stretch RE1 mobile manipulator as the primary robot. For HRI capabilities, an external microphone is mounted on the base of the robot. Additionally, an iPad is mounted on the head of the robot for displaying the robot\u2019s face. Our overall software architecture comprises several different subsystems operating in tandem, coordinated by 4 different layers of finite- state machines. Level 1: A high-level mission planner coordinates the global status of tasks that are executed. Level 2: Each subsystem has its own local FSM that is used to execute sub-tasks. Level 3: Our manipulation subsystem has several complex algorithms that require their own FSMs executing within Level 2 FSMs. Level 4: All of our subsystems are orchestrated by action servers which are based on a low-level FSM.<\/p>\n\n\n\n

Our major subsystems are described below:<\/p>\n\n\n\n

\n\n\n\n

Mission Planner<\/h2>\n\n\n\n

The Mission Planner subsystem enables the integration of the three primary subsystems of our robot: HRI, Navigation, and Manipulation. The Mission Planner manages the current state of the system through a finite state machine (FSM). This FSM triggers different sub-systems of the robot to be activated and perform their part in the larger task. Additionally, all the perception-related computation was performed on a remote GPU device, called the Brain. For low-level robot control, we have developed our own wrapper around the Stretch RE1\u2019s low-level controller, called Alfred Driver. The various architectures governing the Mission Planner\u2019s decision-making and the Brain \/ Alfred Driver\u2019s work are shown in Figures 1 and 2.<\/p>\n\n\n

\n
\"\"
Fig. 1: Mission Planner Architecture<\/strong><\/figcaption><\/figure>\n<\/div>\n\n
\n
\"\"
Fig. 2: Alfred driver and Alfred brain architecture<\/strong><\/figcaption><\/figure>\n<\/div>\n\n\n

We use a single script in order to launch our entire robot in a coordinated manner. When launched, our custom software automatically ensures robot calibration, runs system checks to ensure all sensors are operating nominally, and ensures that all ROS nodes are running nominally before the robot is ready to receive commands.<\/p>\n\n\n\n

\n\n\n\n

Human-Robot Interaction<\/h2>\n\n\n
\n
\"\"
Fig. 3: HRI subsystem architecture <\/strong><\/figcaption><\/figure>\n<\/div>\n\n\n

The above figure shows the HRI subsystem architecture in a flowchart. The Human-Robot Interaction subsystem (HRI) enables the Speech Engagement<\/strong> and Telepresence capabilities<\/strong> of the robot. The Speech Engagement subsystem was implemented with Picovoice for trigger word detection, Google Cloud\u2019s speech-to-text API for speech parsing, and DeepMind\u2019s Neural2 API for life-like voice synthesis. Additionally, ChatGPT was integrated into the HRI subsystem to handle social engagement requests. The trigger-word detector is always running in the background. When the user says the trigger word, the speech-parsing API is activated. The user\u2019s subsequent speech is then parsed and the mission planner activates the respective subsystem to perform the desired task. If the user\u2019s request is identified as a pick-and-place command, the mission planner performs task allocation to appropriately identify the desired object to be placed. If the user\u2019s request is identified as a social-engagement command, the mission planner offloads the task to ChatGPT to generate a response. Finally, the robot gives feedback to the user through a speech command which appropriately addresses the user\u2019s request.<\/p>\n\n\n\n

[Add cropped video of HRI part with help from atharva]@apusalka<\/p>\n\n\n\n

The HRI subsystem\u2019s telepresence capabilities include teleoperation using a handheld interface UI and video-calling implemented with the Agora API. For enabling wireless communication for video calling and teleoperation, the Firebase cloud service is used. Additionally, an iPad is mounted on the robot\u2019s head to display the robot\u2019s face, as well as a video-calling screen. The HRI subsystem\u2019s functioning can be observed in figures below.<\/p>\n\n\n

\n
\"\"
Fig. 4: Multiple iterations of design for interactive eyes of Alfred<\/strong><\/figcaption><\/figure>\n<\/div>\n\n
\n
\"\"
Fig. 5: Video calling feature<\/strong><\/figcaption><\/figure>\n<\/div>\n\n\n

<\/kbd> <\/p>\n\n\n\n

Fig. 6: Teleoperation test<\/strong><\/p>\n\n\n\n

\n\n\n\n

Navigation<\/h2>\n\n\n\n

The navigation subsystem is responsible for moving the robot in the operating environment from an initial location to the desired location of the object, and back to the initial location (where the object is to be placed). It is implemented in Python and C++ using the ROS Noetic platform. At a high level, the navigation subsystem performs functions such as Mapping, Localization, Motion Planning, Dynamic Obstacle Avoidance, Controls, and Actuation. The overall architecture of the navigation subsystem is shown below.<\/p>\n\n\n

\n
\"\"
Fig. 7: Navigation subsystem architecture<\/strong><\/figcaption><\/figure>\n<\/div>\n\n\n

The development of the navigation subsystem started with the generation of a 2D map of the AI Maker Space (test environment). This was generated using SLAM with the help of the gmapping package in ROS. Given that the position of the Stretch RE1\u2019s 2D Lidar is at knee-height, several static objects in the environment, such as tables, were not included in the initial map generated. In order to prevent the robot from planning paths through these regions, the 2D map was manually edited using a software called GIMP. Specifically, regions, where the robot should not traverse, were marked as unknown regions (grey color) in the map of Fig 8.<\/p>\n\n\n

\n
\"\"
Fig. 8: Figure showing map of AI makerspace and Alfred avoiding obstacles<\/strong><\/figcaption><\/figure>\n<\/div>\n\n\n

For localization, we used the Adaptive Monte Carlo Localization algorithm, implemented in the amcl package in ROS Noetic. This algorithm would use the robot\u2019s real-time odometry and lidar data for the prediction and update steps, respectively. For robot path planning and controls, the movebase ROS library was used. We used Dijkstra\u2019s algorithm (Navfn ROS) for global path planning and the Dynamic Window Approach (DWA) algorithm for local path planning and controls. While these planners usually generated feasible paths through the test environment, there were times when the robot would struggle to navigate around an obstacle. In these situations, the robot would trigger its recovery behaviors. The recovery behaviors used by our navigation subsystem were Clear Costmap Recovery, Rotate Recovery, and Moveback Recovery. The Moveback Recovery behavior is a custom plugin developed by the team that commands the robot to move back by a set distance and replan a path, in cases where the other two recovery behaviors fail. Additionally, we extensively tuned ROS parameters in Costmap, Planner, and AMCL packages to optimize the robot\u2019s performance in the test environment. An image depicting the robot avoiding obstacles during navigation is shown in Figure 8.<\/p>\n\n\n\n

3D Navigation<\/strong>: Besides 2D map-based navigation, we also implemented Real-Time Appearance-Based Mapping (RTAB Map). It is a graph-based SLAM approach based on an incremental appearance-based loop closure detector. Fig. 9 shows a 3D map created after running this SLAM approach in a simulation environment. This approach is less susceptible to navigation across thin objects in comparison to 2d navigation since it uses additional modality (RGBD camera) to create a map instead of just a 2D LiDAR. We didn’t deploy this approach for spring validation demonstration due to time constraints and test time problems.<\/p>\n\n\n

\n
\"\"
Fig. 9: 3D map created using RTAB-Map algorithm in simulation<\/strong><\/figcaption><\/figure>\n<\/div>\n\n\n
\n\n\n\n

Manipulation<\/h2>\n\n\n\n

The goal of the manipulation subsystem is to successfully pick the desired object and place at the desired destination once the navigation is completed. Our manipulation subsystem handles both pick as well as place requests. Once a pick request is issued to the subsystem, we execute tasks in sequence as described in our architecture in Figure 10.<\/p>\n\n\n

\n
\"\"
Fig. 10: Manipulation subsystem architecture and Visual Servoing Algorithm<\/strong><\/figcaption><\/figure>\n<\/div>\n\n\n

The detailed working of manipulation subsystem can be defined in the following steps.<\/p>\n\n\n\n