ML Security PapersBona fide Cross Testing Reveals Weak Spot in Audio Deepfake Detection Systems
Chin Yuen Kwok, Jia Qi Yip, Zhen Qiu et al. · Nanyang Technological University
Proposes bona fide cross-testing evaluation framework revealing overlooked vulnerabilities in audio deepfake detection systems across diverse speech types
Audio deepfake detection (ADD) models are commonly evaluated using datasets that combine multiple synthesizers, with performance reported as a single Equal Error Rate (EER). However, this approach disproportionately weights synthesizers with more samples, underrepresenting others and reducing the overall reliability of EER. Additionally, most ADD datasets lack diversity in bona fide speech, often featuring a single environment and speech style (e.g., clean read speech), limiting their ability to simulate real-world conditions. To address these challenges, we propose bona fide cross-testing, a novel evaluation framework that incorporates diverse bona fide datasets and aggregates EERs for more balanced assessments. Our approach improves robustness and interpretability compared to traditional evaluation methods. We benchmark over 150 synthesizers across nine bona fide speech types and release a new dataset to facilitate further research at https://github.com/cyaaronk/audio_deepfake_eval.