Deepfake detectors are DUMB: A benchmark to assess adversarial training robustness under transferability constraints

Deepfake detection systems deployed in real-world environments are subject to adversaries capable of crafting imperceptible perturbations that degrade model performance. While adversarial training is a widely adopted defense, its effectiveness under realistic conditions -- where attackers operate with limited knowledge and mismatched data distributions - remains underexplored. In this work, we extend the DUMB -- Dataset soUrces, Model architecture and Balance - and DUMBer methodology to deepfake detection. We evaluate detectors robustness against adversarial attacks under transferability constraints and cross-dataset configuration to extract real-world insights. Our study spans five state-of-the-art detectors (RECCE, SRM, XCeption, UCF, SPSL), three attacks (PGD, FGSM, FPBA), and two datasets (FaceForensics++ and Celeb-DF-V2). We analyze both attacker and defender perspectives mapping results to mismatch scenarios. Experiments show that adversarial training strategies reinforce robustness in the in-distribution cases but can also degrade it under cross-dataset configuration depending on the strategy adopted. These findings highlight the need for case-aware defense strategies in real-world applications exposed to adversarial attacks.

Key Contributions

• Extends the DUMB/DUMBer evaluation methodology to deepfake detection, enabling systematic robustness assessment under transferability and cross-dataset constraints.
• Comprehensive evaluation of five state-of-the-art deepfake detectors (RECCE, SRM, XCeption, UCF, SPSL) against three adversarial attacks (PGD, FGSM, FPBA) across two datasets (FaceForensics++, Celeb-DF-V2).
• Finds that adversarial training improves in-distribution robustness but can degrade cross-dataset performance depending on strategy, motivating case-aware defenses.

🛡️ Threat Analysis

Details

Similar Papers