Arm Visual Servoing for Button Pressing:

We utilize Inverse-Dynamics to control the upper body (the arms). The system ensures a good precision and accuracy.

Fig: Robot Pressing a Button with Language Prompt.

Fig. Visual servo arm pipeline.

Door Opening Task:

We regard door opening as a whole-body loco-manipulation task and handle it using a trained joint-level control policy with Reinforcement Learning (RL). The simulation environment is built in IsaacSim.

The policy is optimized to open slightly damped doors with a crash bar. It takes in proprioceptions (to capture humanoid’s state) and processed perceptions data (to capture door’s state) and outputs the desired joint positions of 29 joints. A PD controller is then implemente to eventually transfer the joint position to torques.

To alleviate the sim2real gap, we further create the door model in simulator to match the real door.

Fig : Simulation Environment: The humanoid initializing in front of the modeled door.