Privacy-Preserving End-to-End Full-Duplex Speech Dialogue Models

End-to-end full-duplex speech models feed user audio through an always-on LLM backbone, yet the speaker privacy implications of their hidden representations remain unexamined. Following the VoicePrivacy 2024 protocol with a lazy-informed attacker, we show that the hidden states of SALM-Duplex and Moshi leak substantial speaker identity across all transformer layers. Layer-wise and turn-wise analyses reveal that leakage persists across all layers, with SALM-Duplex showing stronger leakage in early layers while Moshi leaks uniformly, and that Linkability rises sharply within the first few turns. We propose two streaming anonymization setups using Stream-Voice-Anon: a waveform-level front-end (Anon-W2W) and a feature-domain replacement (Anon-W2F). Anon-W2F raises EER by over 3.5x relative to the discrete encoder baseline (11.2% to 41.0%), approaching the 50% random-chance ceiling, while Anon-W2W retains 78-93% of baseline sBERT across setups with sub-second response latency (FRL under 0.8 s).

Key Contributions

• First empirical characterization of speaker identity leakage in E2E full-duplex LLM speech models (SALM-Duplex, Moshi) showing leakage persists across all transformer layers with near-perfect attacker EER as low as 6.4% for Moshi
• Layer-wise and turn-wise leakage analysis revealing Linkability rises sharply within the first few dialogue turns, posing GDPR compliance risks
• Two streaming anonymization setups (Anon-W2W waveform-level and Anon-W2F feature-domain) that raise EER from 11.2% to 41.0% while retaining 78–93% dialogue utility at sub-second latency

🛡️ Threat Analysis

Details

Similar Papers