Vision-based Elevator Floor Detection:

The pipeline detects and reads the elevator floor indicator from camera images in a frame-by-frame manner. First, the input image is spatially cropped to reduce background noise, typically keeping the top half and central region where the panel is likely to appear. A vision-language model (VLM) is then used to localize the floor indicator based on a semantic query, producing a bounding box for the display.

To improve stability, detections across consecutive frames are temporally smoothed using simple overlap-based matching and averaging. The resulting bounding box is then expanded slightly and passed to a dedicated readout stage.

In the readout stage, the cropped region is processed using an HSV-based mask to extract illuminated pixels from the display. A percentile-based binarization is applied to enhance contrast, followed by a tighter crop around the foreground. The processed image is then fed into an OCR model with a restricted character set to obtain the final floor symbol. The output is formatted as structured data together with the detection result for downstream use.

Fig: The Pipeline of Elevator Floor Detection.

IMU-based Elevator Floor Detection:

The imu based elevator floor detection uses the z-direction acceleration from imu to detect spikes, and then split the floor transition based on the time interval between a pair of spikes.

Fig: A demonstration of spike detection and time duration distribution.

Button Detection:

The button pose detection pipeline uses a VLM-based vision module to detect elevator button panels from the live camera stream and produce a 6-DoF
target pose (position and orientation). The detected pose is transformed through the robot’s frame chain (camera → base → arm) so the arm controller can servo toward the button. Stability guards reject single-frame false positives and hold the last valid pose briefly to prevent oscillation, making the
detection repeatable enough for downstream arm control to execute button presses reliably.

Fig: Button Detection Results in Elevator with Language Prompt (for both in and out elevator).

Door State Detection:

A ROS 2 action server was developed to detect elevator door states by monitoring real-time point cloud density changes within a tracked region. The system leverages a vision-language model to localize and track the elevator door, then calibrates baseline point count and depth over an initial observation window. During operation, incoming point clouds are depth-filtered around the calibrated region to remove noise, and a smoothed percentage metric—based on deviations from the baseline—is used to infer door state. A configurable threshold determines whether the door is open or closed, while robustness is ensured through startup grace periods, data timeout handling, thread-safe processing, and enforcement of single active monitoring sessions.

Fig: A Visualization of Door State Detection.