ML Security PapersPASTA: A Patch-Agnostic Twofold-Stealthy Backdoor Attack on Vision Transformers
Dazhuang Liu 1, Yanqi Qiao 1, Rui Wang 1, Kaitai Liang 2,1, Georgios Smaragdakis 1
Published on arXiv
2604.20047
Model Poisoning
OWASP ML Top 10 — ML10
Key Finding
Achieves 99.13% attack success rate across arbitrary patch locations while improving visual stealthiness by 144.43x, attention stealthiness by 18.68x, and robustness against defenses by 2.79x compared to state-of-the-art baselines
PASTA
Novel technique introduced
Vision Transformers (ViTs) have achieved remarkable success across vision tasks, yet recent studies show they remain vulnerable to backdoor attacks. Existing patch-wise attacks typically assume a single fixed trigger location during inference to maximize trigger attention. However, they overlook the self-attention mechanism in ViTs, which captures long-range dependencies across patches. In this work, we observe that a patch-wise trigger can achieve high attack effectiveness when activating backdoors across neighboring patches, a phenomenon we term the Trigger Radiating Effect (TRE). We further find that inter-patch trigger insertion during training can synergistically enhance TRE compared to single-patch insertion. Prior ViT-specific attacks that maximize trigger attention often sacrifice visual and attention stealthiness, making them detectable. Based on these insights, we propose PASTA, a twofold stealthy patch-wise backdoor attack in both pixel and attention domains. PASTA enables backdoor activation when the trigger is placed at arbitrary patches during inference. To achieve this, we introduce a multi-location trigger insertion strategy to enhance TRE. However, preserving stealthiness while maintaining strong TRE is challenging, as TRE is weakened under stealthy constraints. We therefore formulate a bi-level optimization problem and propose an adaptive backdoor learning framework, where the model and trigger iteratively adapt to each other to avoid local optima. Extensive experiments show that PASTA achieves 99.13% attack success rate across arbitrary patches on average, while significantly improving visual and attention stealthiness (144.43x and 18.68x) and robustness (2.79x) against state-of-the-art ViT defenses across four datasets, outperforming CNN- and ViT-based baselines.
Key Contributions
- Discovery of Trigger Radiating Effect (TRE) where patch-wise triggers activate backdoors across neighboring patches in ViTs
- Multi-location trigger insertion strategy enabling backdoor activation at arbitrary patch positions during inference
- Bi-level optimization framework for adaptive backdoor learning that balances attack effectiveness with visual and attention stealthiness
🛡️ Threat Analysis
Core contribution is a backdoor/trojan attack on Vision Transformers with trigger-based activation. The paper introduces multi-location trigger insertion during training to embed hidden malicious behavior that activates with specific patch-wise triggers at inference time.