ML Security PapersEnvironmental Injection Attacks against GUI Agents in Realistic Dynamic Environments
Yitong Zhang 1, Ximo Li 1, Liyi Cai 2, Jia Li 1
Published on arXiv
2509.11250
Input Manipulation Attack
OWASP ML Top 10 — ML01
Prompt Injection
OWASP LLM Top 10 — LLM01
Key Finding
Chameleon significantly outperforms existing EIA methods on six realistic websites against four representative LVLM-powered GUI agents under dynamic environment conditions where prior attacks largely fail.
Chameleon
Novel technique introduced
Graphical User Interface (GUI) agents are increasingly deployed to interact with online web services, yet their exposure to open-world content renders them vulnerable to Environmental Injection Attacks (EIAs). In these attacks, an attacker can inject crafted triggers into website to manipulate the behavior of GUI agents used by other users. In this paper, we find that most existing EIA studies fall short of realism. In particular, they fail to capture the dynamic nature of real-world web content, often assuming that a trigger's on-screen position and surrounding visual context remain largely consistent between training and testing. To better reflect practice, we introduce a realistic dynamic-environment threat model in which the attacker is a regular user and the trigger is embedded within a dynamically changing environment. Under this threat model, existing approaches largely fail, suggesting that their effectiveness in exposing GUI agent vulnerabilities has been substantially overestimated. To expose the hidden vulnerabilities of existing GUI agents effectively, we propose Chameleon, an attack framework with two key novelties designed for dynamic environments. (1) To synthesize more realistic training data, we introduce LLM-Driven Environment Simulation, which automatically generates diverse, high-fidelity webpage simulations that mimic the variability of real-world dynamic environments. (2) To optimize the trigger more effectively, we introduce Attention Black Hole, which converts attention weights into explicit supervisory signals. This mechanism encourages the agent to remain insensitive to irrelevant surrounding content, thereby improving robustness in dynamic environments. We evaluate Chameleon on six realistic websites and four representative LVLM-powered GUI agents, where it significantly outperforms existing methods.
Key Contributions
- Introduces a realistic dynamic-environment threat model for EIAs where the attacker is a regular web user and triggers are embedded in dynamically changing web content, exposing flaws in prior static-assumption evaluations.
- Proposes LLM-Driven Environment Simulation to auto-generate diverse, high-fidelity webpage simulations for more realistic trigger training data.
- Proposes Attention Black Hole, which uses VLM attention weights as explicit supervisory signals to make triggers robust to irrelevant surrounding dynamic content.
🛡️ Threat Analysis
Triggers are adversarially optimized using attention weights (Attention Black Hole) as supervisory signals to manipulate VLM perception at inference time — this is adversarial content manipulation of an LLM-integrated system, analogous to adversarial document injection for RAG, warranting dual ML01+LLM01 tagging per the multimodal attack rule.