Benchmarking Adversarial Patch Selection and Location

Adversarial patch attacks threaten the reliability of modern vision models. We present PatchMap, the first spatially exhaustive benchmark of patch placement, built by evaluating over 1.5e8 forward passes on ImageNet validation images. PatchMap reveals systematic hot-spots where small patches (as little as 2% of the image) induce confident misclassifications and large drops in model confidence. To demonstrate its utility, we propose a simple segmentation guided placement heuristic that leverages off the shelf masks to identify vulnerable regions without any gradient queries. Across five architectures-including adversarially trained ResNet50, our method boosts attack success rates by 8 to 13 percentage points compared to random or fixed placements. We publicly release PatchMap and the code implementation. The full PatchMap bench (6.5B predictions, multiple backbones) will be released soon to further accelerate research on location-aware defenses and adaptive attacks.

Key Contributions

PatchMap: a public dataset of 100M+ location-conditioned adversarial patch predictions covering all possible placements on 50K ImageNet validation images across multiple patch sizes and architectures
Systematic spatial analysis exposing stable hot-spots where patches as small as 2% of the image induce confident misclassifications, with unified ASR and confidence-drop metrics across five model architectures
Segmentation-guided zero-gradient placement heuristic using off-the-shelf masks that boosts attack success rate by 8–13 percentage points over random or fixed baselines, including on adversarially trained ResNet-50

🛡️ Threat Analysis

Input Manipulation Attack

Adversarial patches are canonical input manipulation attacks at inference time; this paper benchmarks their spatial effectiveness and proposes a segmentation-guided placement heuristic that measurably improves attack success rates across five architectures.