ML Security PapersFit for Purpose? Deepfake Detection in the Real World
Guangyu Lin , Li Lin , Christina P. Walker , Daniel S. Schiff , Shu Hu
Published on arXiv
2510.16556
Output Integrity Attack
OWASP ML Top 10 — ML09
Key Finding
All evaluated deepfake detectors — academic, government, and industry — fail to generalize to real-world political deepfakes, with top AUC reaching only 74.78% for images and 73.67% for videos, and all systems vulnerable to simple manipulations
The rapid proliferation of AI-generated content, driven by advances in generative adversarial networks, diffusion models, and multimodal large language models, has made the creation and dissemination of synthetic media effortless, heightening the risks of misinformation, particularly political deepfakes that distort truth and undermine trust in political institutions. In turn, governments, research institutions, and industry have strongly promoted deepfake detection initiatives as solutions. Yet, most existing models are trained and validated on synthetic, laboratory-controlled datasets, limiting their generalizability to the kinds of real-world political deepfakes circulating on social platforms that affect the public. In this work, we introduce the first systematic benchmark based on the Political Deepfakes Incident Database, a curated collection of real-world political deepfakes shared on social media since 2018. Our study includes a systematic evaluation of state-of-the-art deepfake detectors across academia, government, and industry. We find that the detectors from academia and government perform relatively poorly. While paid detection tools achieve relatively higher performance than free-access models, all evaluated detectors struggle to generalize effectively to authentic political deepfakes, and are vulnerable to simple manipulations, especially in the video domain. Results urge the need for politically contextualized deepfake detection frameworks to better safeguard the public in real-world settings.
Key Contributions
- First systematic benchmark for deepfake detection using the Political Deepfakes Incident Database — real-world political deepfakes circulating on social media since 2018
- Comprehensive evaluation of academic, government, and industry deepfake detectors revealing poor generalization to authentic real-world content
- Identification that all evaluated detectors are vulnerable to simple manipulations, with paid tools outperforming free-access models but still failing to generalize
🛡️ Threat Analysis
Paper directly evaluates AI-generated content detection systems (deepfake detectors) on real-world political synthetic media, introduces a new benchmark dataset for measuring detector output integrity, and identifies systematic failures including vulnerability to simple manipulations.