Performance

Spring Validation Demonstration

Vision Subsystem:

The vision subsystem performance is evaluated in the above experiment setup. To easily measure the ground truth and the prediction value, we fixed the COBORG on a frame. The depth camera (D435i) is on the left shoulder pointing straight to the board. To validate the consistency of the performance, 10 predefined positions on two different part textures (i.e. shiny, matte) are prepared on the board to test. Figure 1 shows the experiment setting for vision system validation testing. These 10 predefined positions cover different moving directions of hands and different distances between two hands. Table 1 gives the summary of the SVD results on vision system.

Voice Subsystem

The results in Table 1 are from an isolated validation test with the microphone mounted on the COBORG using Gerald’s voice for the test. 150 voice commands were tested including the command to go to the user’s goal target, the command to compact to the arm, and the emergency stop command. A successful output was defined as an accurate recognition of the intended command in at most 2 tries. This criterion was defined based on irritation, if a command takes more than 2 tries to recognize then it irritates the user and defeats the purpose of convenience of voice control. After each trial of three commands, a randomly generated sentence of at least 20 words was read to test if the voice system would detect the keyword “COBORG” trigger in random speech. The goal of this test is to validate the 60% recognition accuracy dictated by requirement P.M.4.1.

Actuated Manipulation Subsystem:

Before SVD, the actuated manipulation subsystem was tested on 3 pre-measured positions on a vertical board directly in front of the robot arm distanced about 90 centimeters away from the robot frame. Each of the 3 pre-measured positions was tested a total of 6 times. The robot arm was commanded to go to these goal positions from the compact/home position. The robot arm was allowed to pass through two intermediate positions between the compact/home positions and the goal position. When the robot arm reached the goal position, a tape measure was used to measure the error distance between the ground truth pre-measured point and the landed position of the end effector on the board. The end effector has to be firmly pressed onto the board. The robot arm was when commanded to go back to the compact/home positions. The robot arm passed through the same two intermediate positions before reaching the compact/home position. Once the robot arm reached the position, a measurement was taken with a tape measure between the base motor and the end effector tip of the robot arm.

The setup image above shows the 3 pre-measured goal positions that were tested on the COBORG during SVD. Position 02 was set orthogonally to the right from Position 01 by 15 centimeters. Position 03 was set orthogonally downward from Position 02 by 15 centimeters.

Results Summary:

Fall Validation Demonstration

System Integration Testings:

For the Fall Validation Demonstration, the COBORG system was required to hold and stabilize a representative part while an operator used a drill and screws to attach the part to a forward-facing and overhead wall. The user also detached the board from these walls. Additionally, the user was allowed no more than five minutes for each task.  During FVD, the system successfully attached and detached the panel in both the forward-facing and overhead scenarios; these tasks took the user a little over a minute to complete, well within the allotted five minutes.  A human demonstration was also given for comparison, which took only slightly less time than the COBORG system.  For FVD Encore, the weight of the panels was increased from 3.5 lbs to 7.5 lbs and even 15 lbs.  The 7.5 lb panel was successfully attached and detached in the forward-facing and overhead scenarios.  The 15 lb panel was only tested in the forward-facing configuration for safety considerations.  It was successfully attached to the wall, but some slipping necessitated the user re-aligning the arm during detachment.  More statistics on the system’s success rate can be found in Table 2 above.

Another requirement for the system was that it would support typical usage for at least 20 minutes.  At the end of both FVD and FVD Encore, the total battery life of the system was extrapolated and indicated an entire lifespan of more than 2 hours, well over the required 20 minutes.

The comfort of the system was tested in an informal manner. The system was expected to be worn for 30 consecutive minutes without discomfort. After both FVD and FVD Encore, the user, who had been wearing the backpack for even longer, was asked to report on the device’s comfort.  The user remarked that the device was surprisingly snug and comfortable, with no issues standing out to them.  The user had also been wearing the COBORG backpack for multiple hours consecutively leading up to the demos.

The voice subsystem proved successful by our requirements by the end of spring, so the only changes before FVD were adding final commands and ROS publishers and subscribers for improved interfacing. Because the functionality was the same as the spring, no quantitative tests were used to validate the voice subsystem for FVD or FVD Encore. Instead, the voice subsystem was implicitly tested and validated through successful demonstration of the total integrated use case of the COBORG, with the voice subsystem serving as the control interface for the user to the robot. We tested extreme cases with the voice recognition subsystem during these demonstrations by proving successful functionality in environments with lots of background voices and with loud manufacturing plant background noise during FVD Encore. We faced some challenges during the FVD Encore when our original system microphone became unresponsive as the audience entered the room. Luckily, we had planned for this in risk analysis and had a backup microphone to hot-swap. However, the new microphone was not performing with the same sensitivity and directionality, causing some recognition issues during the demonstration. We also proved voice recognition capabilities during the poster presentation with our original microphone despite the background noise of many other people speaking and asking questions nearby. Through these passive trials, we implicitly validated the success of the voice recognition system integrated in the COBORG.