ML Security PapersSpeech DF Arena: A Leaderboard for Speech DeepFake Detection Models
Sandipana Dowerah 1, Atharva Kulkarni 2, Ajinkya Kulkarni 3, Hoan My Tran 4, Joonas Kalda 1, Artem Fedorchenko 1, Benoit Fauve 5, Damien Lolive 6, Tanel Alumäe 1, Matthew Magimai Doss 3
Published on arXiv
2509.02859
Output Integrity Attack
OWASP ML Top 10 — ML09
Key Finding
Most evaluated SOTA systems exhibit high equal error rates in out-of-domain scenarios, revealing that current audio deepfake detectors lack robust cross-domain generalization.
Speech DF Arena
Novel technique introduced
Parallel to the development of advanced deepfake audio generation, audio deepfake detection has also seen significant progress. However, a standardized and comprehensive benchmark is still missing. To address this, we introduce Speech DeepFake (DF) Arena, the first comprehensive benchmark for audio deepfake detection. Speech DF Arena provides a toolkit to uniformly evaluate detection systems, currently across 14 diverse datasets and attack scenarios, standardized evaluation metrics and protocols for reproducibility and transparency. It also includes a leaderboard to compare and rank the systems to help researchers and developers enhance their reliability and robustness. We include 14 evaluation sets, 12 state-of-the-art open-source and 3 proprietary detection systems. Our study presents many systems exhibiting high EER in out-of-domain scenarios, highlighting the need for extensive cross-domain evaluation. The leaderboard is hosted on Huggingface1 and a toolkit for reproducing results across the listed datasets is available on GitHub.
Key Contributions
- First comprehensive benchmark (Speech DF Arena) evaluating audio deepfake detection across 14 diverse datasets with standardized metrics (EER, pooled EER, accuracy, F1) for reproducibility
- Hosted leaderboard on HuggingFace covering 12 open-source and 3 proprietary SOTA detection systems with a reproducible evaluation toolkit on GitHub
- Empirical finding that most SOTA detectors exhibit high EER in out-of-domain scenarios, exposing a critical generalization gap between lab benchmarks and real-world conditions
🛡️ Threat Analysis
Audio deepfake detection is canonically ML09 (detecting AI-generated content / output integrity). This paper establishes a standardized evaluation benchmark for exactly this threat — measuring how well detection systems identify synthetic or voice-converted speech across diverse attack scenarios and datasets.