AWARE

Project Overview

Unrecongized anomalies in critical infrastructure can rapidly cascade into catastrophic physical or operational failures. Current anomaly detection approaches often rely on direct access to the monitored system by utilizing onboard sensors, such as motor encoders, to detect anomalous states. However, legacy hardware often lacks these sensors, or strict security protocols may prohibit access altogether.

In this work, we present Adaptive World-model for Anomaly REcognition (AWARE), a world model framework that is capable of anomaly detection using only external camera monitoring via system dynamics estimation and state prediction.

Evaluated across 240 real-world robotic crane trajectories, including payload collisions and motor degradations, AWARE demonstrates a 26% improvement in anomaly classification over our strongest baseline under noisy, camera-only state estimation. Furthermore, the inferred dynamics parameter can be decoded to identify the source of faults, localizing the degrading motor with 88% accuracy.

Methodology

AWARE features two major components:

• Latent estimator (also the latent) that estimates internal dynamics parameters (e.g. armature, center of mass) from a history window of states and actions.

• Adaptive world model (also the predictor) that is conditioned on dynamic latents aside from states and actions, such that it learns to give acuurate state predictions even under circumstances when dynamics parameters deviate.

In the meantime, AWARE also provides accurate estimates of modelled dynamic parameters, allowing localization of the issue and running further introspection.

AWARE is trained fully in simulation. When initializing, the system is first calibrated with live recordings of nominal trajectories, which later serve as reference. During deployment, we compute independent ANOMALY SCORES of both predictor and latent signals via a Mahalanobis Distance measurement. When either ANOMALY SCORES exceed its calibrated threshold, AWARE flags that as an anomaly and alerts the operator, enabling proactive intervention before catastrophic events occur.

Deployment Specs

The entire system uses CCTV camera feeds as single source of truth to estimate both states and operator actions. Unlike modern VLMs which usually require large VRAMs and run with high latency, AWARE is designed to operate in real-time (20Hz), fully on device (2 NVIDIA Jetson Orin), and consume less than 200W power.

Results

We evaluate AWARE across 240 real-world robotic crane trajectories, covering a mixture of payload collisions and motor degradations scenarios. We demonstrate

• 77% detection success rate for payload collisions and motor degradation using high noise, camera-based state estimation.
• highly accurate world model prediction, with about 3 degrees of error over a 2 second horizon.
• 96% success rate in detecting which motor is impaired with only camera-based state estimation.

Limitations

A core limitation of the AWARE method is its assume fixed operating conditions under our reference distribution. Whilst AWARE is robust to the initial distribution choice, moving to a new non-anomalous operating point (e.g. a mobile robot traversing varying terrain) may trigger false positives. Future work could address this through periodic updates of the reference distribution or defining separate intrinsic and extrinsic reference distributions to separate harmless changes in the environment from true intrinsic failures.

Extensive Work

We design AWARE to be platform-agnostic. It would be interesting to see the transferability of method on a complete different platform, for example, a quadruped robot. The robot contains higher dimensional observations that poses many challenges to the state estimation, such as self-occlusion. In the meantime, the moving robot suggests a dynamic nominal condition that changes with respect to the interection between robot and the environment. Also, it remains unclear whether AWARE still detects reliably when the locotion policy internally incorporates a level of self-correction.

Citation

@article{bold2026aware,
  author  = {Luke Beddow, George Mavroghenis, Rodrigo Alonso Chacon Quesada, Kamil Dreczkowski, Cong Sun, Jiankai Wang, Oscar Kwong-Fai Pang, Antoine Cully},
  title   = {Adaptive World Model for Anomaly Recognition},
  journal = {Conference on Robot Learning},
  year    = {2026}
}