{"id":156,"date":"2026-04-03T20:32:17","date_gmt":"2026-04-04T01:32:17","guid":{"rendered":"https:\/\/mrsdprojects.ri.cmu.edu\/2026teama\/?page_id=156"},"modified":"2026-04-24T14:05:04","modified_gmt":"2026-04-24T19:05:04","slug":"perception-system","status":"publish","type":"page","link":"https:\/\/mrsdprojects.ri.cmu.edu\/2026teama\/perception-system\/","title":{"rendered":"Perception Sub-system"},"content":{"rendered":"\n

Vision-based Elevator Floor Detection:<\/strong><\/p>\n\n\n\n

The pipeline detects and reads the elevator floor indicator from camera images in a frame-by-frame manner. First, the input image is spatially cropped to reduce background noise, typically keeping the top half and central region where the panel is likely to appear. A vision-language model (VLM) is then used to localize the floor indicator based on a semantic query, producing a bounding box for the display.<\/p>\n\n\n\n

To improve stability, detections across consecutive frames are temporally smoothed using simple overlap-based matching and averaging. The resulting bounding box is then expanded slightly and passed to a dedicated readout stage.<\/p>\n\n\n\n

In the readout stage, the cropped region is processed using an HSV-based mask to extract illuminated pixels from the display. A percentile-based binarization is applied to enhance contrast, followed by a tighter crop around the foreground. The processed image is then fed into an OCR model with a restricted character set to obtain the final floor symbol. The output is formatted as structured data together with the detection result for downstream use.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Fig:<\/strong> The Pipeline of Elevator Floor Detection.<\/strong><\/p>\n\n\n\n

<\/p>\n\n\n\n

<\/p>\n\n\n\n

IMU-based Elevator Floor Detection:<\/strong><\/p>\n\n\n\n

The imu based elevator floor detection uses the z-direction acceleration from imu to detect spikes, and then split the floor transition based on the time interval between a pair of spikes.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Fig: A demonstration of spike detection and time duration distribution.<\/strong><\/p>\n\n\n\n

<\/p>\n\n\n\n

Button Detection:<\/strong><\/p>\n\n\n\n

The button pose detection pipeline uses a VLM-based vision module to detect elevator button panels from the live camera stream and produce a 6-DoF
target pose (position and orientation). The detected pose is transformed through the robot’s frame chain (camera \u2192 base \u2192 arm) so the arm controller can servo toward the button. Stability guards reject single-frame false positives and hold the last valid pose briefly to prevent oscillation, making the
detection repeatable enough for downstream arm control to execute button presses reliably.<\/p>\n\n\n\n

\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n<\/figure>\n\n\n\n

Fig: Button Detection Results in Elevator with Language Prompt (for both in and out elevator). <\/strong><\/p>\n\n\n\n

<\/p>\n\n\n\n

Door State Detection:<\/strong><\/p>\n\n\n\n

A ROS 2 action server was developed to detect elevator door states by monitoring real-time point cloud density changes within a tracked region. The system leverages a vision-language model to localize and track the elevator door, then calibrates baseline point count and depth over an initial observation window. During operation, incoming point clouds are depth-filtered around the calibrated region to remove noise, and a smoothed percentage metric\u2014based on deviations from the baseline\u2014is used to infer door state. A configurable threshold determines whether the door is open or closed, while robustness is ensured through startup grace periods, data timeout handling, thread-safe processing, and enforcement of single active monitoring sessions.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n

Fig: A Visualization of Door State Detection.<\/strong><\/p>\n\n\n\n

<\/p>\n\n\n\n

<\/p>\n\n\n\n

<\/p>\n\n\n\n

<\/p>\n\n\n\n

<\/p>\n\n\n\n

<\/p>\n\n\n\n

<\/p>\n","protected":false},"excerpt":{"rendered":"

Vision-based Elevator Floor Detection: The pipeline detects and reads the elevator floor indicator from camera images in a frame-by-frame manner. First, the input image is spatially cropped to reduce background noise, typically keeping the top half and central region where the panel is likely to appear. A vision-language model (VLM) is then used to localize […]<\/p>\n","protected":false},"author":434,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-156","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2026teama\/wp-json\/wp\/v2\/pages\/156","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2026teama\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2026teama\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2026teama\/wp-json\/wp\/v2\/users\/434"}],"replies":[{"embeddable":true,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2026teama\/wp-json\/wp\/v2\/comments?post=156"}],"version-history":[{"count":7,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2026teama\/wp-json\/wp\/v2\/pages\/156\/revisions"}],"predecessor-version":[{"id":222,"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2026teama\/wp-json\/wp\/v2\/pages\/156\/revisions\/222"}],"wp:attachment":[{"href":"https:\/\/mrsdprojects.ri.cmu.edu\/2026teama\/wp-json\/wp\/v2\/media?parent=156"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}