Spring Validation Demo System Performance
Our overall system performance for Spring Validation Demo and Encore is as shown below:
| Performance Requirement | Performance Evaluation | |
| PR1 | The system shall receive and respond to the user within 5 seconds | achieved: <= 2 seconds |
| PR2 | The system shall localize in the pre-mapped environment within 5 cm | achieved: within 10 cm |
| PR3 | The system shall collect medical supplies with a success rate of over 80% | achieved: 84% |
| PR4 | The system shall plan its path in less than 5 seconds and navigate to its destination with a maximum speed of 0.3 m/s | achieved: 0.6 m/s |
| PR5 | The system shall deliver medical supplies to the operating room within 5 minutes | achieved: 2 minutes |
| PR6 | The system shall inspect 10 unique items with an accuracy of 2 item counts for the LOW segment | achieved: 100% accuracy via weight-based inspection |
| PR7 | The system shall update the global inventory knowledge base at most every 12 hours | achieved: real-time inventory knowledge |
Fall Validation Demo System Performance
Overall, the system demonstrated consistent and reliable performance during testing, meeting the expectations set for its operation. While there were areas that could benefit from further optimization, the results generally validate the robustness and efficiency of the system in handling the tasks it was designed for.
The table below summarizes our system performance for Fall Validation Demo and Encore:
| Test Case | Required Success Rate | Required Time | Total tests Conducted | Successful Tests | Success Rate | Average Time |
| Delivery/Retrieval | 80% | 5 min | 20 | 16 | 80% | 3 min |
| Restocking | 80% | 3 min | 50 | 48 | 96% | 2 min |
| Combined (4 restocking + 1 delivery) | 80% | 23 min | 12 | 11 | 92% | 12 min |
Overall Subsystem Performance
| Subsystem | Description | Success rate |
| Suction Pose Generation | Success rate of generating feasible suction pose | 88% |
| Motion Planning | Success rate of planning a path to a given pose without collision | 90% |
| Scene Understanding | Success rate of detecting all items in a scene | 80% |
| Navigation | Success rate of reaching a goal pose within an error bound of 5cm | 95% |
We have tested our subsystem-level pipeline rigorously, all achieving 80%+ success rates, as depicted above.