Perception
The goal of the perception subsystem is to determine the location of nearby cornstalks and whether the cornstalks detected are suitable for sensor insertion. This is split up into two tasks: coarse detection –the estimation of the location of the cornstalks relative to the camera– and fine detection –the width measurement of multiple angles of the cornstalk to determine optimal insertion angle. These two tasks are described further below.
Coarse detection is the same as the work done by Aaron Berger and Mark (Moonyang) Lee for stalk detection. First, an image of the environment is captured by an Intel Realsense D405 RGB-D camera. Then the masks of the cornstalks are estimated using a MaskRCNN model. Multiple feature points along the length of the cornstalk are sampled in the depth image, then a line is fit to the cornstalk in 3D space using RANSAC. Then, the detected cornstalks are filtered by whether they are reachable, multi-detections are merged, and the best cornstalk is selected based on distance from the manipulator, and detection confidence. This location of the cornstalk can then be determined in the global frame using a transform.
Fine detection is the same process as coarse detection, but after the masks are detected, the major axis of the cornstalk is detected, and the number of pixels perpendicular to this axis are totaled. The median of the pixels in each row is taken, then the depth of the cornstalk is used to determine the width in real world units. This process takes place multiple times while the camera orbits around the cornstalk so that the side of maximum width can be used for sensor insertion.
Manipulation
The goal of the manipulation subsystem is to facilitate the maintenance of sensors for successful insertions within cornstalks. The subsystem provides the connection between the perception, external mechanisms, and end-effector subsystems.
For planning, we chose to leverage the capabilities of the xArm API and the xarm_ros package. This allowed for easy and robust planning for our application further allowing us for faster planning times.
The xArm goes through a series of trajectories while being heavily integrated into the system Finite State Machine (FSM). The xArm first moves to the home position, and positions the gripper to look down at the base of the cornstalk row, to identify the nearest cornstalk for insertion.
Upon retrieving the nearest cornstalk’s grasp points from the FSM, the xArm orbits around the cornstalk at a fixed radius to identify the best angle for a suitable insertion. The system allows for the nitrate sensor to interface with the External Mechanisms subsystem for clearing and calibration. With the nitrate sensor extended, the xArm moves to each nozzle as required by the FSM, to complete the calibration routine. The xArm proceeds to hook the cornstalk, such that the grasp point is well positioned within the gripper of the End-Effector and adjusts to face the best angle of insertion. Once the insertion has been completed, the xArm moves back to the home position to continue the cycle by identifying the next nearest cornstalk in the waypoint.
End Effector
The primary purpose of the end effector subsystem is to facilitate [1] the gripping of a selected cornstalk and [2] the insertion and removal of a nitrate sensor into a selected cornstalk. The main components that enabled these functionalities were a 50 mm linear actuator, a spring-loaded V-centering block (housed within the electronics housing box), and a rigid V-shaped gripper. The positioning and shaping of the latter two components were carefully crafted to facilitate successful gripping and sensor insertion/removal on cornstalks of 15 mm to 45 mm major-axis length (note: cornstalk is of ovular cross-section). The spring-loaded V-centering block is actuated with the 50 mm linear actuator to clasp the cornstalk. The spring mechanism allows for all forces to be propagated toward sensor insertion (once the springs are compressed, the V-centering block is pushed back, and the nitrate sensor is exposed).
The secondary purpose of the end effector subsystem is to expose the nitrate sensor appropriately for the cleaning and calibration processes. For this purpose, there is a carefully crafted through-hole at which the sensor is fixed with respect to the spring-loaded V-centering block. Looking at this through-hole from a top-down view, both nodes of the nitrate sensor are exposed. This allows for the solutions to drip onto the nitrate sensor and appropriately clean or calibrate it.
In addition to these functionalities, the end effector subsystem also satisfies the following:
- Houses the nitrate sensor’s data logger.
- Houses the D405 in-arm camera, which is used for cornstalk width detection.
- Protects all end-effector electronics from the liquids used in the nitrate sensor’s cleaning and calibration processes.
External Mechanisms
The goal of the external mechanisms subsystem is to implement methods for cleaning, calibrating, and drying, ensuring that data obtained from sensor insertions are reliable. This semester, our focus was on the development of the cleaning and calibration subsystems through the implementation of a downward dripping mechanism. This mechanism encompasses 3 peristaltic pumps, tubing, a microcontroller, 3 relays, a PCB, and computing resources. Upon receiving the initiation signal, the pumps activate sequentially, triggering the cleaning and calibration processes. This approach guarantees the sensor is thoroughly cleaned and calibrated before insertion, enhancing the quality and precision of the data collected.
Navigation
The objective of the Navigation System is to guide the Amiga base through rows of a cornfield. This system comprises two main sub-systems: Global Navigation and Local Navigation.
Global Navigation involves guiding the Amiga robot to follow a trajectory generated from GPS coordinates. This task is executed using a Model Predictive Controller.
Local Navigation entails maneuvering the Amiga robot between two waypoints. This sub-system is essential as the robot explores the waypoint region to locate multiple cornstalks, requiring it to orient itself accordingly. This objective is achieved through visual servoing utilizing a depth camera mounted on the Amiga base.