ML Security PapersSpeechForensics: Audio-Visual Speech Representation Learning for Face Forgery Detection
Yachao Liang 1,2, Min Yu 1,2, Gang Li 3, Jianguo Jiang 1,2, Boquan Li 4, Feng Yu 1, Ning Zhang 5, Xiang Meng 1,2, Weiqing Huang 1,2
Published on arXiv
2508.09913
Output Integrity Attack
OWASP ML Top 10 — ML09
Key Finding
Outperforms state-of-the-art face forgery detection methods on cross-dataset generalization and robustness benchmarks without using any fake videos during model training.
SpeechForensics
Novel technique introduced
Detection of face forgery videos remains a formidable challenge in the field of digital forensics, especially the generalization to unseen datasets and common perturbations. In this paper, we tackle this issue by leveraging the synergy between audio and visual speech elements, embarking on a novel approach through audio-visual speech representation learning. Our work is motivated by the finding that audio signals, enriched with speech content, can provide precise information effectively reflecting facial movements. To this end, we first learn precise audio-visual speech representations on real videos via a self-supervised masked prediction task, which encodes both local and global semantic information simultaneously. Then, the derived model is directly transferred to the forgery detection task. Extensive experiments demonstrate that our method outperforms the state-of-the-art methods in terms of cross-dataset generalization and robustness, without the participation of any fake video in model training. Code is available at https://github.com/Eleven4AI/SpeechForensics.
Key Contributions
- Self-supervised masked audio-visual speech prediction task that learns representations encoding both local and global semantic information from real videos
- Zero-shot transfer of audio-visual speech representations to face forgery detection without requiring any fake video during training
- Demonstrates improved cross-dataset generalization and robustness to common perturbations over prior SOTA deepfake detectors
🛡️ Threat Analysis
Proposes a novel deepfake/face forgery detection method — AI-generated content detection is a core ML09 concern. The contribution is a new forensic detection architecture (self-supervised masked audio-visual prediction then transfer to forgery detection), not a mere domain application of existing detectors.