On Optimizing Multimodal Jailbreaks for Spoken Language Models

As Spoken Language Models (SLMs) integrate speech and text modalities, they inherit the safety vulnerabilities of their LLM backbone and an expanded attack surface. SLMs have been previously shown to be susceptible to jailbreaking, where adversarial prompts induce harmful responses. Yet existing attacks largely remain unimodal, optimizing either text or audio in isolation. We explore gradient-based multimodal jailbreaks by introducing JAMA (Joint Audio-text Multimodal Attack), a joint multimodal optimization framework combining Greedy Coordinate Gradient (GCG) for text and Projected Gradient Descent (PGD) for audio, to simultaneously perturb both modalities. Evaluations across four state-of-the-art SLMs and four audio types demonstrate that JAMA surpasses unimodal jailbreak rate by 1.5x to 10x. We analyze the operational dynamics of this joint attack and show that a sequential approximation method makes it 4x to 6x faster. Our findings suggest that unimodal safety is insufficient for robust SLMs. The code and data are available at https://repos.lsv.uni-saarland.de/akrishnan/multimodal-jailbreak-slm

Key Contributions

• JAMA framework combining GCG (text) and PGD (audio) for joint multimodal jailbreak optimization
• Demonstrates 1.5x-10x higher jailbreak success rates compared to unimodal attacks across 4 state-of-the-art SLMs
• Sequential approximation method (SAMA) achieving 4x-6x speedup with comparable attack success rates

🛡️ Threat Analysis

Details

Similar Papers