Performance

Spring Validation Experiment

Experiment: Acquire at least 3 items from shelf bins and drop in the order bin in under 20 minutes.

Description: A shelf will be setup with 25 items distributed over different shelf bins (1-3 items per shelf, no occlusion). The PR2 based assistive robot should autonomously grasp items off the shelf and drop them in the order bin.

Location: NSH – B level

Equipment: UR5, Kinect,Custom UR5 end effector and Kinect mount,UR5 robot and support structure,UR5 interface, laptop, shelf, items with different geometries (shortlisted from APC 2015 item dictionary), order bin, gripper subsystem – ShopVac, suction cup based end-effecter, Arduino Nano based suction PCB, pressure sensor

1. Populate shelf with all 25 items from the 2015 APC item dictionary
2. Input text file indicating the 10 randomly selected items as well as bin locations
3. The system will…
   1. Automatically recognize items in the bin and report results to a GUI on the computer
   2. Automatically detect object and recognize its pose to find a valid suction surface
   3. Automatically move the arm to the desired grasping location
   4. Attempt to grasp the item without damaging or dropping it
   5. Withdraw the end effector / item from the shelf bin and place it into the order bin
4. The system will repeat the delivery process for 20 minutes and attempt to deliver as many items into the order bin as possible

Results

While testing, we had a 61% success rate with 147 successful picks and 97 failed picks. Items such as the plush puppies squeaky toy, pencils, bunny book, scotch bubble mailer had really good success rates as they had distinct textures and easily graspable surfaces. The table below shows the high level overview of our performance.

System failures are broken down in figure 19. A majority of our failures were due to identification errors. It is difficult to correctly identify items if they are occluded. Identification failure occurred between items with similar textures such as the 40w light bulb and dove soap. Suction and grasping failure occurs for objects such as the DVD, joke book and duct tape. Also, some items such as the water bottle, glue sticks and command hooks were specular – creating sparse point clouds. This made it difficult to compute correct grasp surfaces. The figures below show our failure analysis statistics.

Item by Item failure analysis

Fall Validation Experiments

During the fall validation experiment, all targeted requirements were met. In addition, during the fall validation experiment encore, we transitioned from running in simulation to running the system on PR2. Tabulation above highlights the outcome of the Fall Validation Experiments.

Experiment 1: Perception Experiment

Description: A setup will be made with shelf bin containing 1-3 items (no occlusion) and the perception system outputs the shelf location of the item of interest

Location: NSH – B level

Equipment: Kinect, Laptop, Shelf bin, items with different geometries (shortlisted from APC 2015 item dictionary)

1. Station Kinect opposite to shelf; shelf position with respect to Kinect is known
2. Randomly place 1 item on the shelf, non-occluding.
3. Run perception through command line
4. Output 3D scene w/ bounding box around item of interest
5. Repeat steps 2-4 four times
6. Repeat steps 2-5 for 2 items and 3 items on the shelf, record 50% success.

Experiment Results

Experiment 2: State Control and Hardware in the Loop Grasping Experiment

Description: A shelf bin will be setup with a random assortment of items. The gripper subsystem should autonomously grasp items off the shelf and drop them in the order bin. The arm should move between the shelf and order bin in simulation.

Location: NSH – B level

Equipment: Complete Gripper Subsystem – ShopVac, suction cup based gripper end-effector, Arduino Nano based suction PCB, desktop computer

1. Professor Dolan randomly selects and distribute up to 3 items on the real world shelf bin
2. State controller is initiated, arm moves to the shelf bin in simulation
3. The state controller enters the grasping state and the vacuum turns on in the real world
4. The item is manually acquired in the real world by an external operator holding the suction gripper, approached from the top.
5. The pressure sensor sense a seal is formed in the real world
6. The robot arm moves from the shelf to the order bin in simulation
7. The vacuum turns off when the robot reaches the order bin; the item is dropped in the real world
8. Repeat steps 3-8 for 10 items, recording 70% success

Experiment Results

The video below shows the results of the initial fall validation experiment. The PR2 was able to access all shelf bins and control the gripper hardware.

The video below shows the results of the second fall validation experiment. We were able to execute all PR2 actions on the SBPL PR2.

Spring Test Plan