{"id":46,"date":"2024-11-25T22:07:08","date_gmt":"2024-11-25T22:07:08","guid":{"rendered":"https:\/\/mrsdprojects.ri.cmu.edu\/2025teama\/?page_id=46"},"modified":"2025-12-12T13:02:17","modified_gmt":"2025-12-12T13:02:17","slug":"system-implementation","status":"publish","type":"page","link":"https:\/\/mrsdprojects.ri.cmu.edu\/2025teama\/system-implementation\/","title":{"rendered":"System Implementation"},"content":{"rendered":"\n

Full System Fall Semester Implementation<\/h2>\n\n\n\n

Full System Diagram<\/h3>\n\n\n\n
\"\"<\/figure>\n\n\n\n

This is the system we use for Fall Validation Demonstrations.<\/p>\n\n\n\n

Each subsystem implementation detail, including hardware, sensing, perception, planning, Software, UI, and autonomy, is in the following pages.<\/p>\n\n\n\n\n\n\n\n

Hardware<\/h2>\n\n\n\n

Wheelchair<\/h3>\n\n\n\n

The wheelchair serves as the primary platform for the autonomous mobility system. It integrates various sensors and computational units to enable autonomous navigation, teleoperation, and real-time perception. The wheelchair can successfully carry payload over 80KG and run at 1m\/s.<\/p>\n\n\n\n

Seat<\/h3>\n\n\n\n

We select a lawn mower seat that features a thick cushion and a built-in seat belt and armrest. We further mount this seat on two aluminum extrusion pieces. This design effectively transfers the weight to the platform, enabling it to support a weight capacity of up to 200 pounds.<\/p>\n\n\n\n

Footrest<\/h3>\n\n\n\n

We constructed the footrest using nine aluminum extrusions, securely mounted onto the wheelchair platform to enhance structural rigidity and overall durability.<\/p>\n\n\n

\n
\"\"<\/figure>\n<\/div>\n\n\n

Jetson Orin<\/h3>\n\n\n\n
    \n
  • Installed an NVIDIA Jetson Orin module as the primary compute unit<\/li>\n\n\n\n
  • Configured wireless connectivity by setting up an M.2 wireless card, ensuring reliable communication with the ROS ecosystem and external controllers.<\/li>\n<\/ul>\n\n\n
    \n
    \"\"<\/figure>\n<\/div>\n\n\n

    RGBD Camera<\/h3>\n\n\n\n
      \n
    • Mounted a ZED X Camera onto the wheelchair platform, ensuring a stable and vibration-free setup.<\/li>\n<\/ul>\n\n\n
      \n
      \"\"<\/figure>\n<\/div>\n\n\n

      Dual Antenna RTK-GPS<\/h3>\n\n\n\n
        \n
      • Two Antenna RTK-GPS are mounted on the wheelchair platform for better stability<\/li>\n<\/ul>\n\n\n
        \n
        \"\"<\/figure>\n<\/div>\n\n\n

        Powertrain<\/h3>\n\n\n\n
          \n
        • 12 V LiFePO\u2084 battery, motor controller, and electric motor wired over CAN<\/li>\n\n\n\n
        • Triple hardwired E-STOP buttons plus wireless kill-switch (ESP32 + CC1101 PCB) with < 100 ms response<\/li>\n<\/ul>\n\n\n
          \n
          \"\"<\/figure>\n<\/div>\n\n\n

          Subsystem Diagram<\/h3>\n\n\n\n
          \"\"<\/figure>\n\n\n\n\n\n\n\n

          Sensing<\/h2>\n\n\n\n

          RGBD Camera<\/h3>\n\n\n\n
            \n
          • Getting 30 fps RGB images, 10 fps point clouds, odometry<\/li>\n<\/ul>\n\n\n
            \n
            \"\"<\/figure>\n<\/div>\n\n\n

            Dual Antenna RTK-GPS<\/h3>\n\n\n\n
              \n
            • Connecting to AirLab\u2019s RTK base station\n
                \n
              • Precise location with error less than 1m<\/li>\n\n\n\n
              • Accurate heading from dual antenna and IMU<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n
                \n
                \"\"<\/figure>\n<\/div>\n\n\n

                Subsystem Diagram<\/h3>\n\n\n\n
                \"\"<\/figure>\n\n\n\n\n\n\n\n

                Perception<\/h2>\n\n\n\n

                Cost-mapping<\/h3>\n\n\n\n
                  \n
                • Collected our own expert data (20 x 2mins), and trained a new Maxent IRL model on it<\/li>\n\n\n\n
                • Cost map (20m x 20m) updated in 2~3 hz<\/li>\n<\/ul>\n\n\n\n
                  \"\"<\/figure>\n\n\n\n

                  Curb Detection<\/h3>\n\n\n\n
                    \n
                  • Run in 1 hz on the wheelchair<\/li>\n<\/ul>\n\n\n\n
                    \"\"<\/figure>\n\n\n\n

                    Subsystem Diagram<\/h3>\n\n\n\n
                    \"\"<\/figure>\n\n\n\n\n\n\n\n

                    Planning<\/h2>\n\n\n\n

                    Waypoint Generation<\/h3>\n\n\n\n

                    Integrated OpenStreetMap for waypoints on sidewalks.<\/p>\n\n\n\n

                      \n
                    • Prevented car-road goal waypoints in favor of walkable paths.<\/li>\n<\/ul>\n\n\n\n
                      \"\"<\/figure>\n\n\n\n

                      VLM Planner<\/h3>\n\n\n\n

                      The VLM planner option is integrated in planning pipeline<\/p>\n\n\n\n

                        \n
                      • Enabled single user-command intermediate waypoint planning.<\/li>\n\n\n\n
                      • Enabled human-in-the-loop interactive planning.<\/li>\n<\/ul>\n\n\n\n
                        \"\"<\/figure>\n\n\n\n

                        Subsystem Diagram<\/h3>\n\n\n\n
                        \"\"<\/figure>\n\n\n\n\n\n\n\n

                        Software<\/h2>\n\n\n\n

                        Visualization<\/h3>\n\n\n\n

                        Foxglove<\/h4>\n\n\n\n

                        Foxglove offers a better visualization than rviz in both real-time and ROS bag<\/p>\n\n\n\n

                        \"\"<\/figure>\n\n\n\n

                        Data Collection Setup<\/h3>\n\n\n\n

                        RGBD Camera<\/h4>\n\n\n\n
                          \n
                        • Successfully connected and synchronized the ZED X camera with ROS.<\/li>\n\n\n\n
                        • Recorded depth and RGB data streams into a ROS bag for offline processing and model training.<\/li>\n\n\n\n
                        • Custom ROS 2 intra-process recorder (IPC-based) for high-throughput bag capture (\u2265 5 fps) \u200b<\/li>\n<\/ul>\n\n\n\n
                          \"\"<\/figure>\n\n\n\n

                          Ouster LiDAR<\/h4>\n\n\n\n
                            \n
                          • Established communication with the Ouster OS1 LiDAR through ROS.<\/li>\n\n\n\n
                          • Captured real-time 3D point cloud data and recorded it into a ROS bag.<\/li>\n<\/ul>\n\n\n\n
                            \"\"<\/figure>\n\n\n\n\n\n\n\n

                            User Interface<\/h2>\n\n\n\n

                            Functionality<\/h3>\n\n\n\n
                              \n
                            • Remote control interface.<\/li>\n\n\n\n
                            • Integrates the Google Navigation API for global route planning.<\/li>\n\n\n\n
                            • Includes speech-to-text functionality.\n
                                \n
                              • Converts user voice input into ROS 2 messages.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                              • Working on a user selection interface for user feedback.<\/li>\n<\/ul>\n\n\n\n

                                iOS Application<\/h3>\n\n\n\n

                                Developed an integrated UI for teleoperation and navigation.<\/p>\n\n\n

                                \n
                                \"\"<\/figure>\n<\/div>\n\n\n

                                Teleoperation<\/h4>\n\n\n\n
                                  \n
                                • Established ROS2 communication between the iOS app and the wheelchair.<\/li>\n\n\n\n
                                • Implemented fundamental movement controls (forward, backward, left turn, right turn, stop).<\/li>\n\n\n\n
                                • Designed intuitive push-and-hold buttons for continuous movement, ensuring immediate stop upon release for safety.<\/li>\n<\/ul>\n\n\n
                                  \n
                                  \"\"<\/figure>\n<\/div>\n\n\n

                                  Navigation<\/h4>\n\n\n\n
                                    \n
                                  • Destination entry via text or speech; routes fetched from Google Directions, downsampled, and streamed as waypoints<\/li>\n\n\n\n
                                  • Displays raw, filtered, and RTK-corrected GPS plus device orientation; switches fully to RTK after iPhone GPS proved unstable<\/li>\n<\/ul>\n\n\n\n

                                    <\/p>\n\n\n\n

                                    User Interaction<\/h3>\n\n\n\n

                                    Speech to Text <\/h4>\n\n\n\n
                                    \"\"<\/figure>\n\n\n\n

                                    Speech Recognition<\/h4>\n\n\n\n
                                      \n
                                    • Seamless speech-to-text conversion with multi-language support.<\/li>\n\n\n\n
                                    • Use case: Voice commands for setting a destination.<\/li>\n<\/ul>\n\n\n
                                      \n
                                      \"\"<\/figure>\n<\/div>\n\n\n

                                      Smart Assistant<\/h4>\n\n\n\n
                                        \n
                                      • Processing time improvement planned for future updates.<\/li>\n\n\n\n
                                      • Use case: ChatGPT integration for answering various questions.<\/li>\n<\/ul>\n\n\n
                                        \n
                                        \"\"<\/figure>\n<\/div>\n\n\n

                                        Subsystem Diagram<\/h3>\n\n\n\n
                                        \"\"<\/figure>\n\n\n\n\n\n\n\n

                                        Wheelchair Autonomy<\/h3>\n\n\n\n
                                          \n
                                        • RADIO\u2013VLAD feature encoder running at 10 fps on Orin to semantically label terrain<\/li>\n\n\n\n
                                        • Cost-map pipeline fusing RGB features and projected point clouds into a BEV grid<\/li>\n\n\n\n
                                        • Bird\u2019s-eye-view cost-map generation for downstream planning \u200b<\/li>\n\n\n\n
                                        • MPPI planner replanning every 100 ms over the live cost-map<\/li>\n\n\n\n
                                        • Velocity commands sent to the motor controller loop at 10 Hz<\/li>\n<\/ul>\n\n\n
                                          \n
                                          \"\"<\/figure>\n<\/div>\n\n\n

                                          Isaac Sim<\/h3>\n\n\n\n

                                          Wheeled Robot Basic Setup<\/h4>\n\n\n\n
                                            \n
                                          • Full-stack NVIDIA Omniverse Isaac Sim environment mirroring real-world sidewalks, curbs, and obstacles<\/li>\n\n\n\n
                                          • Simulated sensors and MPPI waypoint manager enable end-to-end pipeline testing before field runs<\/li>\n<\/ul>\n\n\n
                                            \n
                                            \"\"<\/figure>\n<\/div>","protected":false},"excerpt":{"rendered":"

                                            Full System Fall Semester Implementation Full System Diagram This is the system we use for Fall Validation Demonstrations. Each subsystem implementation detail, including hardware, sensing, perception, planning, Software, UI, and autonomy, is in the following pages.<\/p>\n","protected":false},"author":391,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"page-templates\/page_fullwidth.php","meta":{"footnotes":""},"class_list":["post-46","page","type-page","status-publish","hentry","clearfix"],"_links":{"self":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2025teama\/wp-json\/wp\/v2\/pages\/46","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2025teama\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2025teama\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2025teama\/wp-json\/wp\/v2\/users\/391"}],"replies":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2025teama\/wp-json\/wp\/v2\/comments?post=46"}],"version-history":[{"count":20,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2025teama\/wp-json\/wp\/v2\/pages\/46\/revisions"}],"predecessor-version":[{"id":419,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2025teama\/wp-json\/wp\/v2\/pages\/46\/revisions\/419"}],"wp:attachment":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2025teama\/wp-json\/wp\/v2\/media?parent=46"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}