Vision – COBORG

Description

Integrated with the whole system and tested the overall performance.
Enable obstacle detection using Octomap with the point cloud from the D435i cameras.
Developed the Goal Getter node to post-process the edge cases and the two camera streams
Integrated two-camera(two D435i) vision system with manipulation
Tested two YOLOv3 running on two camera nodes
Modified the launch file to launch two vision nodes
Optimized original vision workflow to reduce one node from the pipeline
Migrated vision system from x86 to NVIDIA Jetson

Started integration with Actuated Manipulation system.
Vision system development completed!
Post-processing embedded into ROS.
Post-processing on 3D bounding boxes done to get goal position and surface normal.
3D YOLO hand detection embedded into ROS wrapper by using darknet_ros_3d package.
2D YOLOv3 hand detection embedded into ROS wrapper by using darknet_ros package in ROS.
2D YOLOv3 modified to be run through CUDA GPUs. This version was ran in RTX2070, and provided an average 35 FPS.
Intel RealSense ROS API installed onto ROS Melodic framework. Manufacturer-created launch scripts (rs_rgbd.launch) ran to output cameras onto RViz environment.

YOLOv3 implemented onto ROS Melodic framework. This version was ran through CPU power, and provides an average 1 fps on-screen.
Conceptual design made and presented. Subsystem drafted to incorporate Intel RealSense D435i and Intel RealSense T265. D435i will provide RGB video streams and point cloud map. T265 will provide localization and SLAM features output robot motion relative to global frame.
Schedule and technical roadmap draft built for subsystems. These include functional and cyberphysical architectures.
Preliminary requirements built for vision subsystem.
As-Is CoBorg was examined. Developed implementation of vision and perception was on list of must-haves but has not be fully developed yet.