Perception
The goal of the perception subsystem is to determine the location of nearby cornstalks and whether the cornstalks detected are suitable for sensor insertion. This is split up into three tasks: coarse detection –the estimation of the location of the cornstalks relative to the camera, fine detection — the width measurement of multiple angles of the cornstalk to determine optimal insertion angle, and grasp refinement — the final up-close stalk detection to make final adjustments before grasping the stalk. These two tasks are described further below.
Coarse detection is the same as the work done by Aaron Berger and Mark (Moonyang) Lee for stalk detection. First, an image of the environment is captured by an Intel Realsense D405 RGB-D camera. Then the masks of the cornstalks are estimated using a MaskRCNN model. Multiple feature points along the length of the cornstalk are sampled in the depth image, then a line is fit to the cornstalk in 3D space using RANSAC. Then, the detected cornstalks are filtered by whether they are reachable, multi-detections are merged, and the best cornstalk is selected based on distance from the manipulator, and detection confidence. This location of the cornstalk can then be determined in the global frame using a transform.
Fine detection is the same process as coarse detection, but after the masks are detected, the major axis of the cornstalk is detected, and the number of pixels perpendicular to this axis are totaled. The median of the pixels in each row is taken, then the depth of the cornstalk is used to determine the width in real world units. This process takes place multiple times while the camera orbits around the cornstalk so that the side of maximum width can be used for sensor insertion. Grasp refinement also uses the same process as coarse detection, but performs this operation when the camera is much closer to the stalk for a more accurate estimation.
Manipulation
The goal of the manipulation subsystem is to facilitate the maintenance of sensors for successful insertions within cornstalks. The subsystem provides the connection between the perception, external mechanisms, and end-effector subsystems.
For planning, we chose to leverage the capabilities of the MoveIt. This allowed for easy and robust planning for our application further allowing us for faster planning times. MoveIt also allows for the creation of collision boxes which the planner to avoid obstacles in the real world such as the ground, cornstalks, and the Amiga mobile base (shown in the image below).
The xArm goes through a series of trajectories commanded by the Finite State Machine (FSM). The xArm first moves from the stow position to a position where the camera can view the base of the cornstalks to identify the nearest cornstalk for insertion.
Upon retrieving the nearest cornstalk’s grasp points from the FSM, the xArm orbits around the cornstalk at a fixed radius to identify the best angle for a suitable insertion. The system allows for the nitrate sensor to interface with the External Mechanisms subsystem for clearing and calibration. With the nitrate sensor extended, the xArm moves to each nozzle as required by the FSM, to complete the calibration routine. The xArm proceeds to hook the cornstalk, such that the grasp point is well positioned within the gripper of the End-Effector and adjusts to face the best angle of insertion. Once the insertion has been completed, the xArm moves back to the home position to continue the cycle by identifying the next nearest cornstalk in the waypoint.
End Effector
The primary purpose of the end effector subsystem is to facilitate [1] the gripping of a selected cornstalk and [2] the insertion and removal of a nitrate sensor into a selected cornstalk. The main components that enabled these functionalities were a 50 mm linear actuator, a spring-loaded V-centering block (housed within the electronics housing box), and a rigid V-shaped gripper. The positioning and shaping of the latter two components were carefully crafted to facilitate successful gripping and sensor insertion/removal on cornstalks of 15 mm to 45 mm major-axis length (note: cornstalk is of ovular cross-section). The spring-loaded V-centering block is actuated with the 50 mm linear actuator to clasp the cornstalk. The spring mechanism allows for all forces to be propagated toward sensor insertion (once the springs are compressed, the V-centering block is pushed back, and the nitrate sensor is exposed).
The secondary purpose of the end effector subsystem is to expose the nitrate sensor appropriately for the cleaning and calibration processes. For this purpose, there is a carefully crafted through-hole at which the sensor is fixed with respect to the spring-loaded V-centering block. Looking at this through-hole from a top-down view, both nodes of the nitrate sensor are exposed. This allows for the solutions to drip onto the nitrate sensor and appropriately clean or calibrate it.
In addition to these functionalities, the end effector subsystem also satisfies the following:
- Houses the nitrate sensor’s data logger.
- Houses the D405 in-arm camera, which is used for cornstalk width detection.
- Protects all end-effector electronics from the liquids used in the nitrate sensor’s cleaning and calibration processes.
External Mechanisms
The goal of the external mechanisms subsystem is to implement methods for cleaning, calibrating, and drying, ensuring that data obtained from sensor insertions are reliable. This semester, our focus was on the development of the cleaning and calibration subsystems through the implementation of a downward dripping mechanism. This mechanism encompasses 3 peristaltic pumps, tubing, a microcontroller, 3 relays, a PCB, and computing resources. Upon receiving the initiation signal, the pumps activate sequentially, triggering the cleaning and calibration processes. This approach guarantees the sensor is thoroughly cleaned and calibrated before insertion, enhancing the quality and precision of the data collected.
Navigation
The objective of the Navigation System is to guide the Amiga base through rows of a cornfield. This system comprises two main sub-systems: Global Navigation and Local Navigation.
Global Navigation involves guiding the Amiga robot to follow a trajectory generated from GPS coordinates. This task is executed using a Model Predictive Controller.
Local Navigation entails maneuvering the Amiga robot between two waypoints. This sub-system is essential as the robot explores the waypoint region to locate multiple cornstalks, requiring it to orient itself accordingly. This objective is achieved through visual servoing utilizing a depth camera mounted on the Amiga base.
User Interface
The user interface provides the user a means to interact with the system with the specific objectives listed above as shown below. The subsystem was created using a PyQT5 GUI in ROS. The user can input the sampling locations by directly entering the coordinate into the field on the right side of the screen or uploading a CSV file containing a list of coordinates. These coordinates populate in the upper right table with the option to remove unwanted coordinates and representative images of cornstalks appear on the map. The interface also provides messages in the black box if any of the coordinates are invalid.
Once the user chooses to start the system, the image of the Amiga mobile base updates in real time on the map on the left sides of the screen. The base visits each sampling location represented by images of cornstalks. As nitrate readings are collected from each sampling location, the readings populate in the table. Once the system has completed collecting the data from all sampling locations, the readings can be stored in a CSV value containing the readings corresponding to each sampling location.
