Adversarial Examples Are Not Bugs, They Are Superposition

Adversarial examples -- inputs with imperceptible perturbations that fool neural networks -- remain one of deep learning's most perplexing phenomena despite nearly a decade of research. While numerous defenses and explanations have been proposed, there is no consensus on the fundamental mechanism. One underexplored hypothesis is that superposition, a concept from mechanistic interpretability, may be a major contributing factor, or even the primary cause. We present four lines of evidence in support of this hypothesis, greatly extending prior arguments by Elhage et al. (2022): (1) superposition can theoretically explain a range of adversarial phenomena, (2) in toy models, intervening on superposition controls robustness, (3) in toy models, intervening on robustness (via adversarial training) controls superposition, and (4) in ResNet18, intervening on robustness (via adversarial training) controls superposition.

Key Contributions

• Theoretical framework showing superposition can explain six major adversarial example phenomena including transferability, noise-like perturbation structure, and interpretability gains from adversarial training
• Toy model experiments establishing bidirectional causal control between superposition and adversarial robustness
• ResNet18 experiments showing adversarial training reduces superposition as measured by sparse autoencoder (SAE) reconstruction loss

🛡️ Threat Analysis

Details

Similar Papers