Spring 2017

Performance for the Spring-Validation Experiment was determined by comparing the event to the outlined Verification Criteria and the project’s Performance and Non-Functional Requirements. The Verification Criteria was as follows:

1. Pick up at least 12 items and drop them inside their target totes within 15 minutes, dropping no more than 2 items to the floor

2. Drop items into the totes from no more than .3m from the bottom of the totes

3. Generate an item report in the form of a JSON for the items remaining on the shelf, with 100% accuracy for item bin locations (excluding any dropped items)

At the SVE encore demo, we met the Verification Criteria number 2. For Criteria 1, we picked 1 item per bin and successfully dropped them in their target totes. Overall the system successfully picked 4 out of 4 items specified in a total of 6 minutes. No items were dropped to floor during this process. For Criteria 3, the system accurately reported 3 out of 4 of the items in the output JSON file.

Demo video for SVE

Strong points

The planning system has demonstrated robust consistent use of experience graphs and path constraints. This has allowed for much faster and safer generation of pose configuration and path executions during testing and system runs.

 

The perception system has demonstrated that it can accurately identify every item within heavily occluded configurations (20+ items), correctly labeling between 50-95% of each item’s pixels. This consistent high accuracy in classification has been paramount in system success.

Pose generation for grasping is easily configurable, with clear layouts for weighting of features such as centroid, point cloud height, etc. This configurability has made testing and troubleshooting for system runs very fast and informative. Evaluation on grasping hardware is shown below. 

The current state machine design cleanly separates the system logic on a drawer-to-drawer and bin-to-bin basis. This separation allows for consistent patterned strategies for item picking and greatly simplifies the overall system logic.

 

Weak points

The extrinsic calibration between the RGB and Depth portions of the system camera are inaccurate by marginal but relevant amounts. This occasionally results in grasping pose generation locations on the bin walls or floors which miss the target item.

 

The current wooden shelf system causes multiple collision and calibration issues. Small physical interactions between the arm and the shelf, or individuals and the shelf as they place and remove items, can cause very small movements of the shelving system which then affect calibration to the planning scene and produce errors in collision modeling.

 

The arm is currently sensitive to hitting hard torque limits based on the implemented end effector design. This is a product of the end effector weight and length. Currently the design is mostly composed of aluminum, which can be replaced with lighter plastics, and is potentially 1-4 inches longer than it needs to be to still keep optimal system performance.

 

The current feedback method of pressure-sensing based on measurements from the system vacuum has many problems. First, there is an inherent time delay required to get accurate readings because of the length of the vacuum hosing and the time it takes to see pressure differences. Second, the readings are inconsistent and change based off of ambient conditions such as temperature and weather, resulting in misclassifications.

Fall 2016

FVE Evaluation-Gripper Fabrication

For the first Fall Validation Experiment gripper controls were implemented through rosserial. The first FVE demonstrated that the gripper mockup had poor tolerances and allowed for too much play within the pivot for a given angle.

The gripper linkages and suction head were redesigned for the FVE encore to improve tolerance issues. With the new changes the system demonstrated that it could move to any user-specified with a few degrees through ROS.

FVE Evaluation-Perception

The areas of refinement and further work for the spring semester are shown in table 1, below under desired system status column.

The perception system is behind schedule in several areas. The major areas are:

1. Item Segmentation from the Point Cloud must be extended to multiple touching items.
2. The shelf and kinect setup is position invariant. Full pose invariance is preferred, so as to deal with perturbation of the shelf.
3. The item identification needs to be replaced by a more robust model that will account for the surprise items as well.
4. Utilities have to be developed to calibrate and measure the transforms of all kinects as required in the new shelf design.

For FVE, the perception subsystem was integrated with the state machine, however was not conveying any details of item pose to the planning and grasping subsystems. For the FVE Encore the perception system was providing a basic pose estimate to the grasping subsystem that consisted of item centroid and item characteristics.
The item identification for the FVE and FVE Encore was done using a makeshift method, that entailed resizing the cropped item image and feeding it to team HARP’s superpixel classifier to generate the top 5 item predictions. This method was quite unreliable leading to up to 66 percent wrong classifications as it was purely based on dominant item color and thus required the item color to be largely homogenous.

 

FVE Evaluation-Software

The software subsystem was demonstrated to be fully functional in a picking scenario. The end effector pauses in front of each bin to gather point cloud and image data. The system control  moves the end effector to the top of target item based on the height and centroid location supplied by the perception subsystem. Pressure sensors are used to detect a successful grasp. If the pressure sensors are unable to sense a successful grasp the arm keeps moving down until it establishes suction.

 

Due to the stop-motion criteria for the robotic am, results can be disastrous if the state machine doesn’t detect the anticipated pressure sensor reading. Unexpected conditions should be taken into consideration to enhance the robustness of the state machine and allow it to fail “gracefully”.

 

FVE Evaluation-Motion Planning

For FVE, the planning subsystem was fully functional, but had jittery motion while attempting to grasp items. During Encore, this jittering was reduced, but still appeared when the vision system gave an incorrect item pose estimation. The reduction in jitter shaved about one minute off of total test time, which demonstrates the system’s ability to act quickly during competition.

 

Besides jitter, planning acted well during FVE and Encore. Plans were generated quickly and the success of plans was reported accurately. During FVE and Encore, each other subsystem was able to send commands to planning, which demonstrates the desired generic service structure.