ML Security PapersDeepfakeBench-MM: A Comprehensive Benchmark for Multimodal Deepfake Detection
Kangran Zhao 1,2, Yupeng Chen 1,2, Xiaoyu Zhang 1,2, Yize Chen 1,2, Weinan Guan 1,2, Baicheng Chen 1,2, Chengzhe Sun 2, Soumyya Kanti Datta 2, Qingshan Liu 3, Siwei Lyu 2, Baoyuan Wu 1,2
1 The Chinese University of Hong Kong, Shenzhen
Published on arXiv
2510.22622
Output Integrity Attack
OWASP ML Top 10 — ML09
Key Finding
DeepfakeBench-MM unifies evaluation of 11 multimodal deepfake detectors across 5 datasets; Mega-MMDF is one of the largest multimodal deepfake datasets with 1.1M forged samples from 21 forgery pipelines.
DeepfakeBench-MM
Novel technique introduced
The misuse of advanced generative AI models has resulted in the widespread proliferation of falsified data, particularly forged human-centric audiovisual content, which poses substantial societal risks (e.g., financial fraud and social instability). In response to this growing threat, several works have preliminarily explored countermeasures. However, the lack of sufficient and diverse training data, along with the absence of a standardized benchmark, hinder deeper exploration. To address this challenge, we first build Mega-MMDF, a large-scale, diverse, and high-quality dataset for multimodal deepfake detection. Specifically, we employ 21 forgery pipelines through the combination of 10 audio forgery methods, 12 visual forgery methods, and 6 audio-driven face reenactment methods. Mega-MMDF currently contains 0.1 million real samples and 1.1 million forged samples, making it one of the largest and most diverse multimodal deepfake datasets, with plans for continuous expansion. Building on it, we present DeepfakeBench-MM, the first unified benchmark for multimodal deepfake detection. It establishes standardized protocols across the entire detection pipeline and serves as a versatile platform for evaluating existing methods as well as exploring novel approaches. DeepfakeBench-MM currently supports 5 datasets and 11 multimodal deepfake detectors. Furthermore, our comprehensive evaluations and in-depth analyses uncover several key findings from multiple perspectives (e.g., augmentation, stacked forgery). We believe that DeepfakeBench-MM, together with our large-scale Mega-MMDF, will serve as foundational infrastructures for advancing multimodal deepfake detection.
Key Contributions
- Mega-MMDF: a large-scale multimodal deepfake dataset with 1.1M forged samples generated via 21 forgery pipelines (10 audio, 12 visual, 6 audio-driven face reenactment methods)
- DeepfakeBench-MM: the first unified benchmark for multimodal deepfake detection, supporting 5 datasets and 11 detectors with standardized evaluation protocols
- Comprehensive empirical analyses across multiple perspectives (augmentation, stacked forgery) revealing key insights for multimodal deepfake detection
🛡️ Threat Analysis
Deepfake detection is a core ML09 concern — the paper builds infrastructure (dataset + evaluation platform) for detecting AI-generated audiovisual content, directly advancing output integrity and content authenticity research.