The Alignment Curse: Cross-Modality Jailbreak Transfer in Omni-Models

Recent advances in end-to-end trained omni-models have significantly improved multimodal understanding. At the same time, safety red-teaming has expanded beyond text to encompass audio-based jailbreak attacks. However, an important bridge between textual and audio jailbreaks remains underexplored. In this work, we study the cross-modality transfer of jailbreak attacks from text to audio, motivated by the semantic similarity between the two modalities and the maturity of textual jailbreak methods. We first analyze the connection between modality alignment and cross-modality jailbreak transfer, showing that strong alignment can inadvertently propagate textual vulnerabilities to the audio modality, which we term the alignment curse. Guided by this analysis, we conduct an empirical evaluation of textual jailbreaks, text-transferred audio jailbreaks, and existing audio-based jailbreaks on recent omni-models. Our results show that text-transferred audio jailbreaks perform comparably to, and often better than, audio-based jailbreaks, establishing them as simple yet powerful baselines for future audio red-teaming. We further demonstrate strong cross-model transferability and show that text-transferred audio attacks remain effective even under a stricter audio-only access threat model.

Key Contributions

Identifies and theoretically analyzes the 'alignment curse': strong cross-modal alignment inadvertently propagates textual jailbreak vulnerabilities to the audio modality in omni-models
Demonstrates empirically that text-transferred audio jailbreaks match or outperform existing native audio jailbreaks on recent omni-models, establishing them as a simple yet powerful baseline
Shows strong cross-model transferability and effectiveness under a stricter audio-only access threat model