Robust Experts: the Effect of Adversarial Training on CNNs with Sparse Mixture-of-Experts Layers

Robustifying convolutional neural networks (CNNs) against adversarial attacks remains challenging and often requires resource-intensive countermeasures. We explore the use of sparse mixture-of-experts (MoE) layers to improve robustness by replacing selected residual blocks or convolutional layers, thereby increasing model capacity without additional inference cost. On ResNet architectures trained on CIFAR-100, we find that inserting a single MoE layer in the deeper stages leads to consistent improvements in robustness under PGD and AutoPGD attacks when combined with adversarial training. Furthermore, we discover that when switch loss is used for balancing, it causes routing to collapse onto a small set of overused experts, thereby concentrating adversarial training on these paths and inadvertently making them more robust. As a result, some individual experts outperform the gated MoE model in robustness, suggesting that robust subpaths emerge through specialization. Our code is available at https://github.com/KASTEL-MobilityLab/robust-sparse-moes.

Key Contributions

• Demonstrates that inserting a single sparse MoE layer in deeper CNN stages (ResNet) with adversarial training consistently improves robustness under PGD and AutoPGD attacks on CIFAR-100
• Discovers that switch loss-based balancing causes routing collapse onto a small set of experts, inadvertently concentrating adversarial training and creating robust expert subpaths
• Shows that individual overused experts can outperform the full gated MoE model in adversarial robustness, suggesting emergent specialization

🛡️ Threat Analysis

Details

Similar Papers