ATAC: Augmentation-Based Test-Time Adversarial Correction for CLIP

Despite its remarkable success in zero-shot image-text matching, CLIP remains highly vulnerable to adversarial perturbations on images. As adversarial fine-tuning is prohibitively costly, recent works explore various test-time defense strategies; however, these approaches still exhibit limited robustness. In this work, we revisit this problem and propose a simple yet effective strategy: Augmentation-based Test-time Adversarial Correction (ATAC). Our method operates directly in the embedding space of CLIP, calculating augmentation-induced drift vectors to infer a semantic recovery direction and correcting the embedding based on the angular consistency of these latent drifts. Across a wide range of benchmarks, ATAC consistently achieves remarkably high robustness, surpassing that of previous state-of-the-art methods by nearly 50\% on average, all while requiring minimal computational overhead. Furthermore, ATAC retains state-of-the-art robustness in unconventional and extreme settings and even achieves nontrivial robustness against adaptive attacks. Our results demonstrate that ATAC is an efficient method in a novel paradigm for test-time adversarial defenses in the embedding space of CLIP.

Key Contributions

• ATAC: a test-time defense that estimates a semantic recovery direction in CLIP's embedding space using augmentation-induced drift vectors to correct adversarially perturbed image embeddings
• Empirical insight that gradient-based attacks induce consistent directional shifts in CLIP's feature space, enabling reliable recovery without access to labels
• Achieves ~50% improvement in robust accuracy over prior SOTA across 13 classification benchmarks while retaining nontrivial robustness against adaptive attacks

🛡️ Threat Analysis

Details

Similar Papers