AnTrap: Systematic Benchmarking of Android GUI Agent Robustness Under Dynamic Runtime Anomalies

AnTrap is a new benchmark that injects dynamic perturbations into Android GUI agent trajectories, organizing anomalies into a four-layer taxonomy. Evaluating 16 GUI models, the authors report universal vulnerability, with performance degrading for even the strongest models. Adversarial RL training fixes single-step state and action traps but leaves deep contextual traps such as state deadlock unresolved, indicating a limitation of environment-based training.

Hugging Face Daily Papers — Are Android GUI Agents Robust Against Runtime Anomalies? AnTrap: Evaluating Agents in Dynamic Adversarial Environments

August 27, 20261 source80% confidence

74Strategic

Strategic score

Evidence strength60

Technical novelty80

Commercial potential65

AnTrap is a new benchmark for Android GUI agents that systematically injects runtime anomalies into the agent's execution trajectory, moving robustness testing from static, clean-interface benchmarks to dynamic adversarial conditions. Its central claim is that all 16 evaluated GUI models are universally vulnerable to such perturbations, and that the strongest models still suffer significant degradation. The key constraint that has shifted is the evaluation paradigm: rather than asking whether an agent can complete a task, AnTrap asks whether an agent can maintain correct behavior when the interface and execution context change underneath it.¹

Context

Existing GUI-agent benchmarks typically assume a static environment and measure task completion. AnTrap violates that assumption by placing perturbations into four layers—state, thinking, action, and round—and ten subcategories, thus charting the space of failures an agent might encounter in deployment. The construction pipeline is designed to preserve task solvability, which is an important design choice because it isolates the agent's robustness from the environment's feasibility.¹

Assessment

The evidence, as presented in the abstract, is suggestive but not fully assessed: the authors evaluate 16 leading models and report universal degradation, but individual results, effect sizes, and standard deviations are not given. The benchmark's main value lies in its taxonomic decomposition and in the training experiments, which separate anomalies that can be learned away from those that cannot. This is a meaningful distinction, but it rests on the completeness of the taxonomy.¹

Caveats

The claim that deep contextual traps 'cannot be resolved by training in environments with traps alone' is strong; it would be falsified or qualified by evidence that different training algorithms, more data, or different reward shapes do enable agents to escape state deadlocks. Similarly, the universality of the vulnerability is bounded by the set of models tested and the specific perturbation distribution. More evidence, including per-model breakdowns and failure analyses, would change how strongly we can trust the conclusion.¹

Source text

Hugging Face Daily Papers — Are Android GUI Agents Robust Against Runtime Anomalies? AnTrap: Evaluating Agents in Dynamic Adversarial Environments · August 25, 2026

Computer Science > Artificial Intelligence

Title:Are Android GUI Agents Robust Against Runtime Anomalies? AnTrap: Evaluating Agents in Dynamic Adversarial Environments

Abstract:GUI agents often encounter dynamic anomalies when deployed on Android devices, from unexpected pop-ups to action misuse, yet existing benchmarks lack systematic evaluation of agent robustness against runtime anomalies. We introduce AnTrap, a comprehensive benchmark that injects dynamic perturbations into agent execution trajectories. We propose a taxonomy organizing real-world anomalies into four layers (State, Thinking, Action and Round) with ten fine-grained subcategories, and develop a construction pipeline that preserves task solvability while introducing realistic adversarial conditions. Evaluating 16 leading GUI models, we reveal universal vulnerability to dynamic anomalies, with even the strongest models suffering significant performance degradation. Furthermore, we conduct GRPO training in both original and adversarial environments to validate our benchmark, separating environment-learnable anomalies from reasoning-bottlenecked ones. Our findings show that while single-step traps at state and action layers are largely addressable through adversarial reinforcement learning, deep contextual traps, like state deadlock, expose intrinsic limitations that cannot be resolved by training in environments with traps alone.

Access Paper:

Bookmark

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

The abstract’s phrase 'preserves task solvability' carries an unstated assumption: perturbations must not make the task impossible. If a trap could render a state unreachable, then a performance drop would reflect environment feasibility, not lack of agent robustness. The benchmark’s value depends on this constraint, but the abstract does not operationalize it.

GRPO is likely Group Relative Policy Optimization, an on-policy RL method that uses group-wise advantage normalization. The abstract uses it without introducing the term; readers may conflate it with other policy gradient algorithms. The distinction matters because the training results attribute certain robustness failures to the environment itself rather than to the training algorithm.

The term 'state deadlock' is introduced as an example of a deep contextual trap, but the abstract does not define its formal state or dynamic condition. Without a definition, the claim that it exposes 'intrinsic limitations' cannot be independently verified. This is a term-of-art whose precise meaning should be fixed by the full paper.

The source text above remains the property of its publisher and is reproduced for commentary.

Hugging Face Daily Papers — Are Android GUI Agents Robust Against Runtime Anomalies? AnTrap: Evaluating Agents in Dynamic Adversarial Environments

August 25, 2026endistinct angle

Are Android GUI Agents Robust Against Runtime Anomalies? AnTrap: Evaluating Agents in Dynamic Adversarial Environments

↑ Back to top