When and Where to Attack? Stage-wise Attention-Guided Adversarial Attack on Large Vision Language Models

Adversarial attacks against Large Vision-Language Models (LVLMs) are crucial for exposing safety vulnerabilities in modern multimodal systems. Recent attacks based on input transformations, such as random cropping, suggest that spatially localized perturbations can be more effective than global image manipulation. However, randomly cropping the entire image is inherently stochastic and fails to use the limited per-pixel perturbation budget efficiently. We make two key observations: (i) regional attention scores are positively correlated with adversarial loss sensitivity, and (ii) attacking high-attention regions induces a structured redistribution of attention toward subsequent salient regions. Based on these findings, we propose Stage-wise Attention-Guided Attack (SAGA), an attention-guided framework that progressively concentrates perturbations on high-attention regions. SAGA enables more efficient use of constrained perturbation budgets, producing highly imperceptible adversarial examples while consistently achieving state-of-the-art attack success rates across ten LVLMs. The source code is available at https://github.com/jackwaky/SAGA.

Key Contributions

• Empirically establishes that regional attention scores are positively correlated with adversarial loss sensitivity, motivating attention-focused perturbation strategies.
• Discovers that attacking high-attention regions induces structured redistribution of model attention toward subsequent salient regions, enabling a staged attack curriculum.
• Proposes SAGA, a stage-wise attention-guided adversarial attack that progressively concentrates the perturbation budget on high-attention regions, achieving SOTA attack success rates across ten LVLMs with improved imperceptibility.

🛡️ Threat Analysis

Details

Similar Papers