HuLA: Prosody-Aware Anti-Spoofing with Multi-Task Learning for Expressive and Emotional Synthetic Speech

Current anti-spoofing systems remain vulnerable to expressive and emotional synthetic speech, since they rarely leverage prosody as a discriminative cue. Prosody is central to human expressiveness and emotion, and humans instinctively use prosodic cues such as F0 patterns and voiced/unvoiced structure to distinguish natural from synthetic speech. In this paper, we propose HuLA, a two-stage prosody-aware multi-task learning framework for spoof detection. In Stage 1, a self-supervised learning (SSL) backbone is trained on real speech with auxiliary tasks of F0 prediction and voiced/unvoiced classification, enhancing its ability to capture natural prosodic variation similar to human perceptual learning. In Stage 2, the model is jointly optimized for spoof detection and prosody tasks on both real and synthetic data, leveraging prosodic awareness to detect mismatches between natural and expressive synthetic speech. Experiments show that HuLA consistently outperforms strong baselines on challenging out-of-domain dataset, including expressive, emotional, and cross-lingual attacks. These results demonstrate that explicit prosodic supervision, combined with SSL embeddings, substantially improves robustness against advanced synthetic speech attacks.

Key Contributions

• Two-stage prosody-aware multi-task learning framework (HuLA) that first fine-tunes an SSL backbone on F0 prediction and voiced/unvoiced classification using only real speech, then jointly optimizes spoof detection with prosody tasks on real and synthetic data.
• Demonstrates that explicit prosodic supervision (F0 and V-UV structure) substantially improves out-of-domain robustness against expressive, emotional, and cross-lingual synthetic speech attacks.
• Shows that prosodic imperfections in TTS/VC systems serve as a discriminative cue exploitable for spoof detection when combined with SSL embeddings.

🛡️ Threat Analysis

Details

Similar Papers