ML Security PapersViT-EnsembleAttack: Augmenting Ensemble Models for Stronger Adversarial Transferability in Vision Transformers
Hanwen Cao , Haobo Lu , Xiaosen Wang , Kun He
Published on arXiv
2508.12384
Input Manipulation Attack
OWASP ML Top 10 — ML01
Key Finding
ViT-EnsembleAttack significantly outperforms existing ensemble-based transfer attack methods on ViT architectures across extensive experiments.
ViT-EnsembleAttack
Novel technique introduced
Ensemble-based attacks have been proven to be effective in enhancing adversarial transferability by aggregating the outputs of models with various architectures. However, existing research primarily focuses on refining ensemble weights or optimizing the ensemble path, overlooking the exploration of ensemble models to enhance the transferability of adversarial attacks. To address this gap, we propose applying adversarial augmentation to the surrogate models, aiming to boost overall generalization of ensemble models and reduce the risk of adversarial overfitting. Meanwhile, observing that ensemble Vision Transformers (ViTs) gain less attention, we propose ViT-EnsembleAttack based on the idea of model adversarial augmentation, the first ensemble-based attack method tailored for ViTs to the best of our knowledge. Our approach generates augmented models for each surrogate ViT using three strategies: Multi-head dropping, Attention score scaling, and MLP feature mixing, with the associated parameters optimized by Bayesian optimization. These adversarially augmented models are ensembled to generate adversarial examples. Furthermore, we introduce Automatic Reweighting and Step Size Enlargement modules to boost transferability. Extensive experiments demonstrate that ViT-EnsembleAttack significantly enhances the adversarial transferability of ensemble-based attacks on ViTs, outperforming existing methods by a substantial margin. Code is available at https://github.com/Trustworthy-AI-Group/TransferAttack.
Key Contributions
- Three ViT-specific model augmentation strategies (Multi-head dropping, Attention score scaling, MLP feature mixing) with Bayesian-optimized parameters to diversify surrogate ensembles and reduce adversarial overfitting
- Automatic Reweighting and Step Size Enlargement modules to further improve adversarial transferability across ensemble ViTs
- First ensemble-based transfer attack method specifically tailored for Vision Transformers, outperforming existing methods by a substantial margin
🛡️ Threat Analysis
Core contribution is crafting adversarial examples that transfer across ViT architectures via gradient-based optimization on augmented surrogate ensembles — a direct inference-time input manipulation attack.