ML Security PapersAdversarial Attention Perturbations for Large Object Detection Transformers
Zachary Yahn 1, Selim Furkan Tekin 1, Fatih Ilhan 1, Sihao Hu 1, Tiansheng Huang 1, Yichang Xu 1, Margaret Loper 2, Ling Liu 1
Published on arXiv
2508.02987
Input Manipulation Attack
OWASP ML Top 10 — ML01
Key Finding
AFOG outperforms existing adversarial attacks on transformer-based and CNN-based object detectors by up to 83% on COCO while producing visually imperceptible perturbations.
AFOG (Attention-Focused Offensive Gradient)
Novel technique introduced
Adversarial perturbations are useful tools for exposing vulnerabilities in neural networks. Existing adversarial perturbation methods for object detection are either limited to attacking CNN-based detectors or weak against transformer-based detectors. This paper presents an Attention-Focused Offensive Gradient (AFOG) attack against object detection transformers. By design, AFOG is neural-architecture agnostic and effective for attacking both large transformer-based object detectors and conventional CNN-based detectors with a unified adversarial attention framework. This paper makes three original contributions. First, AFOG utilizes a learnable attention mechanism that focuses perturbations on vulnerable image regions in multi-box detection tasks, increasing performance over non-attention baselines by up to 30.6%. Second, AFOG's attack loss is formulated by integrating two types of feature loss through learnable attention updates with iterative injection of adversarial perturbations. Finally, AFOG is an efficient and stealthy adversarial perturbation method. It probes the weak spots of detection transformers by adding strategically generated and visually imperceptible perturbations which can cause well-trained object detection models to fail. Extensive experiments conducted with twelve large detection transformers on COCO demonstrate the efficacy of AFOG. Our empirical results also show that AFOG outperforms existing attacks on transformer-based and CNN-based object detectors by up to 83% with superior speed and imperceptibility. Code is available at https://github.com/zacharyyahn/AFOG.
Key Contributions
- Learnable adversarial attention mechanism that focuses perturbations on vulnerable image regions in multi-box detection tasks, improving over non-attention baselines by up to 30.6%
- Unified attack loss integrating two types of feature loss via learnable attention updates with iterative adversarial perturbation injection, architecture-agnostic across transformers and CNNs
- Empirical evaluation across 12 large detection transformers on COCO showing up to 83% improvement over existing attacks with superior speed and imperceptibility
🛡️ Threat Analysis
AFOG crafts visually imperceptible, gradient-based adversarial perturbations at inference time to cause object detection models (transformers and CNNs) to fail — a textbook Input Manipulation Attack using iterative white-box gradient injection.