SVD Performance

Autonomy Subsystem

For mapping, after generating the map using offline SLAM, we perform preprocessing to ensure structural consistency. The evaluation focuses on verifying that the map does not contain misalignments or outliers that could affect downstream modules.

For initialization, we evaluate the system by placing the robot at different starting positions and measuring both the success rate and response time. The response time is defined as the duration from initialization to the point where the particle filter converges to a stable covariance. As figure shown here, the system demonstrates reliable initialization performance across different conditions.

During runtime, we further evaluate localization stability by monitoring the covariance of the particle filter. As the figure here, the covariance remains below the predefined threshold for most of the operation. Occasional peaks occur in feature-sparse environments, such as corridors, where limited geometric features reduce localization confidence. However, the overall performance remains within acceptable bounds.

For path planning, we evaluate trajectory feasibility using curvature as the primary metric, due to the constraints of the pure pursuit controller and the physical limitations of the platform. The curvature is required to remain below 1.55. We compute discrete curvature along planned trajectories and evaluate performance across different routes. As shown, the 95th percentile curvature remains below the specified threshold, indicating that the generated paths are feasible for execution.

For control performance, we use cross-track error (CTE) as the primary evaluation metric for the pure pursuit controller. Since tracking error tends to be larger during turning maneuvers, we define separate criteria for straight-line and turning scenarios.

Overall, the results indicate that the autonomy subsystem meets the defined performance criteria across all stages of the navigation pipeline, validating both the robustness and feasibility of the system.

User Interface Subsystem

To evaluate the responsiveness of the user interface subsystem, we define performance requirements on system latency and real-time feedback. Specifically, the user interface is required to display robot status in real time, and waypoint publishing must be executed within 50 ms. As shown in Figure, we analyze the execution time of each stage in the communication and processing pipeline, including frontend interaction, backend processing, and ROS2 message transmission. By aggregating the latency contributions from each component, the total end-to-end execution time is measured to be approximately 10.48 ms, which is well below the specified requirement.

Anomaly Detection Subsystem

The anomaly detection subsystem met most of the acceptance criteria defined for the Spring Validation Demonstration (SVD). For the first acceptance criterion, the subsystem was required to successfully detect more than 95\% of non-occluded human skeletons, including joints and connections. We evaluated the OpenPifPaf-based skeleton detection model using multiple test images from both local image inputs and live camera inputs. Based on the calculated precision, recall, and F1 score, the model achieved a precision of 1.00, a recall of 0.944, and an F1 score of 0.971. Since the F1 score exceeded the 95\% threshold, this acceptance criterion was met.

The second acceptance criterion required the generated bounding boxes to accurately enclose valid human subjects. The purpose of this criterion was to ensure that non-human objects, such as desks, tables, or background objects, were not incorrectly enclosed by human bounding boxes. Based on multiple offline and live tests performed during SVD, the generated bounding boxes consistently enclosed valid human subjects, so this criterion was also met.

The third acceptance criterion required the subsystem to re-identify and maintain tracking of a human target when the target was occluded for fewer than 50 frames. This was tested using a downloaded video containing temporary human occlusion. In the test, a person with ID 3 was occluded by another person with ID 2 for fewer than 50 frames. After the occlusion, the same person was successfully re-identified with the original ID 3. Therefore, the subsystem satisfied the re-identification and tracking criterion.

The fourth acceptance criterion was added based on feedback received during SVD. This feature aimed to detect people lying on the ground, as suggested by Knightscope CEO William Santana Li. The detection logic identifies a lying person when the bounding box width is greater than its height. As shown in the test result, when a person is lying on the ground, the subsystem flags the person as “man down” and highlights the skeleton with a red bounding box. This criterion was successfully met.

The final acceptance criterion required the subsystem to correctly flag anomalous actions, such as throwing objects, jumping, pushing, fighting, chasing, loitering, hiding, and playing with a ball, more than 50\% of the time. This criterion was not met. Although the subsystem was able to perform live anomaly detection with some delay, the predicted action labels were not accurate enough. For example, fighting was sometimes classified as falling, and throwing was sometimes classified as chasing. Therefore, the anomaly action classification component still requires significant improvement in both accuracy and latency for the fall semester.