Component Testing and Experiment Results
Our system underwent rigorous component testing to evaluate its performance against our Spring Validation Demonstration (SVD) targets. Across all subsystems, Perception, Planning, Manipulation, and Machine Integration, the system not only met but consistently exceeded our baseline metrics.
Below is a summary of our performance against the established targets, followed by a detailed breakdown of the testing methodologies and results for each subsystem.
Spring Validation Performance Summary
| Task | Target Metric | Achieved Metric | Status |
| PR3a. Recognize Lab Equipment | 90% | 100% | Exceeded |
| PR3b. Localize Lab Equipment | ≤ 3 cm error | < 1 cm error | Exceeded |
| PR4. Plan to Reach Lab Equipment | 95% | 100% | Exceeded |
| PR5. Grasp Lab Equipment | 75% | 100% | Exceeded |
| PR7. Command Lab Machines | 100% | 100% | Met |
Detailed Subsystem Evaluation
Perception
Both detection models were trained and evaluated to surpass our baseline targets of greater than 90% accuracy and fewer than 5% false positives. To ensure robustness, training data were collected across a wide variety of lighting conditions and camera positions.
- Evaluation Metric: We utilized mAP50, which counts a detection as valid if the intersection over union (IoU) of the predicted and ground-truth bounding boxes exceeds 50%.
- Overhead Camera Model: Achieved 99.5% mAP50 on a dataset of 2,000 images (using an 80:20 train/test split).
- Gripper Camera Model: Achieved 99.4% mAP50 on a larger dataset of 21,000 images.
- Key Takeaways: Zero false positives were observed during testing, vastly outperforming our <5% target rate. Furthermore, equipment localization achieved sub-centimeter accuracy (<1 cm error), and AprilTag detection functioned flawlessly.
Planning
The planning subsystem was stress-tested across all conceivable planning scenarios.
- Methodology: We conducted 10 trials per use case, covering pick-and-place operations to and from the OT-2, the shaker module, and the base.
- Results: The system achieved a 100% success rate across all paths, easily surpassing the spring target of 95%.
Manipulation
Grasping and manipulation were evaluated to ensure consistent physical interaction with the lab equipment.
- Methodology: The system attempted 5 grasps at each of the 3 primary locations (the OT-2, the shaker, and the base), resulting in 15 total trials.
- Results: All grasps succeeded (including scenarios requiring retries), yielding a 100% success rate and significantly exceeding our baseline target of 75%.
Lab Machine Integration
The software integration responsible for commanding the lab machines was thoroughly tested alongside the physical systems. Execution was perfect, achieving a 100% success rate.
