ML Security PapersAUDETER: A Large-scale Dataset for Deepfake Audio Detection in Open Worlds
Qizhou Wang 1, Hanxun Huang 1, Guansong Pang 2, Sarah Erfani 1, Christopher Leckie 1
Published on arXiv
2509.04345
Output Integrity Attack
OWASP ML Top 10 — ML09
Key Finding
XLR-based detectors trained on AUDETER achieve 1.87% EER on the In-the-Wild benchmark, demonstrating strong cross-domain generalisation compared to existing detectors trained on prior datasets.
AUDETER
Novel technique introduced
Speech synthesis systems can now produce highly realistic vocalisations that pose significant authenticity challenges. Despite substantial progress in deepfake detection models, their real-world effectiveness is often undermined by evolving distribution shifts between training and test data, driven by the complexity of human speech and the rapid evolution of synthesis systems. Existing datasets suffer from limited real speech diversity, insufficient coverage of recent synthesis systems, and heterogeneous mixtures of deepfake sources, which hinder systematic evaluation and open-world model training. To address these issues, we introduce AUDETER (AUdio DEepfake TEst Range), a large-scale and highly diverse deepfake audio dataset comprising over 4,500 hours of synthetic audio generated by 11 recent TTS models and 10 vocoders, totalling 3 million clips. We further observe that most existing detectors default to binary supervised training, which can induce negative transfer across synthesis sources when the training data contains highly diverse deepfake patterns, impacting overall generalisation. As a complementary contribution, we propose an effective curriculum-learning-based approach to mitigate this effect. Extensive experiments show that existing detection models struggle to generalise to novel deepfakes and human speech in AUDETER, whereas XLR-based detectors trained on AUDETER achieve strong cross-domain performance across multiple benchmarks, achieving an EER of 1.87% on In-the-Wild. AUDETER is available on GitHub.
Key Contributions
- AUDETER: a large-scale deepfake audio dataset of 4,500+ hours and 3M clips synthesized by 11 TTS models and 10 vocoders, addressing gaps in real speech diversity and synthesis system coverage
- Empirical finding that binary supervised training on heterogeneous deepfake sources induces negative transfer, degrading generalisation
- Curriculum-learning-based training strategy to mitigate negative transfer, enabling XLR-based detectors to achieve 1.87% EER on the In-the-Wild benchmark
🛡️ Threat Analysis
Directly addresses AI-generated audio content detection — building a benchmark for evaluating deepfake audio detectors and improving their generalization falls squarely under output integrity and authenticity of AI-generated content.