System-Level State Machine

A robust finite state machine controls high-level decision-making of our system. It starts and terminates the different states in our system as shown in the figure above. The states in orange correspond to the manipulation subsystem, purple are the perception subsystem, green states are the end-effector subsystem, and gray represents manual intervention. Each state in the state machine has two transition conditions. For most states, the transition condition represented by 1 in the diagram indicates a success in executing the current state, and the condition 0 in the diagram represents a failure. In some cases, such as in the approach plant positions state, 0 means that the manipulator has finished finding peppers in all the approach positions, and hence, the system transitions into the end state and stops. This diagram does not account for self-loops as they are handled internally within the particular state.

Our system begins in the reset pose state in which the manipulator moves to a kinematically favorable position. The system moves to an approach position and begins the multi-frame process. The position of the arm moves slightly, and images of the pepper plant are captured at these locations. The cyclic loop between states 2 and 3 in the figure represents this process. The system then requests the perception subsystem to find a point of interaction as indicated by state 4. All the images are processed together to determine the points of interaction of true positive peppers. A chosen pepper’s POI is passed to the manipulation subsystem. 

The arm moves to the pre-grasp POI pose which is offset from the true POI location by a distance along the depth axis. Both the gripper and the cutter are opened to allow the peduncle to be inserted between them. Next, the state machine activates the visual servoing state in the perception subsystem that provides a more accurate location of the POI. The manipulator then moves to a secondary pre-grasp pose informed by the adjustments from visual servoing. Finally, a cartesian move along the depth axis places the end-effector in the correct pose for extraction of the pepper. Our system closes the gripper and the cutter respectively to harvest the chosen pepper. Then, the pepper is dropped off in the basket. Finally, the end-effector is closed again before the arm goes back to an approach position for further identification of peppers. The robot repeatedly returns to the same approach position until no peppers are detected. At this point, it moves to the next approach position. In this manner, our system explores all the approach positions until no peppers are found in any of the positions, upon which the system is terminated. 

GitHub

A link to our GitHub page can be found here. Our GitHub includes repositories with code for each of the major subsystems. The major repositories which we were utilized for the full integrated system include README files with any necessary explanations, configuration instructions, installation instructions, and operating instructions.