ML Security PapersMulti-Faceted Attack: Exposing Cross-Model Vulnerabilities in Defense-Equipped Vision-Language Models
Yijun Yang 1, Lichao Wang 2, Jianping Zhang 1, Chi Harold Liu 2, Lanqing Hong 3, Qiang Xu 1
Published on arXiv
2511.16110
Input Manipulation Attack
OWASP ML Top 10 — ML01
Prompt Injection
OWASP LLM Top 10 — LLM01
Key Finding
MFA achieves 58.5% overall jailbreak success rate across defense-equipped VLMs and 52.8% on commercial models (GPT-4o, Gemini-Pro, Llama-4), outperforming the second-best attack by 34%
Multi-Faceted Attack (MFA) / Attention-Transfer Attack (ATA)
Novel technique introduced
The growing misuse of Vision-Language Models (VLMs) has led providers to deploy multiple safeguards, including alignment tuning, system prompts, and content moderation. However, the real-world robustness of these defenses against adversarial attacks remains underexplored. We introduce Multi-Faceted Attack (MFA), a framework that systematically exposes general safety vulnerabilities in leading defense-equipped VLMs such as GPT-4o, Gemini-Pro, and Llama-4. The core component of MFA is the Attention-Transfer Attack (ATA), which hides harmful instructions inside a meta task with competing objectives. We provide a theoretical perspective based on reward hacking to explain why this attack succeeds. To improve cross-model transferability, we further introduce a lightweight transfer-enhancement algorithm combined with a simple repetition strategy that jointly bypasses both input-level and output-level filters without model-specific fine-tuning. Empirically, we show that adversarial images optimized for one vision encoder transfer broadly to unseen VLMs, indicating that shared visual representations create a cross-model safety vulnerability. Overall, MFA achieves a 58.5% success rate and consistently outperforms existing methods. On state-of-the-art commercial models, MFA reaches a 52.8% success rate, surpassing the second-best attack by 34%. These results challenge the perceived robustness of current defense mechanisms and highlight persistent safety weaknesses in modern VLMs. Code: https://github.com/cure-lab/MultiFacetedAttack
Key Contributions
- Attention-Transfer Attack (ATA) that hides harmful instructions inside a meta task with competing objectives, explained theoretically via reward hacking
- Lightweight transfer-enhancement algorithm combined with a repetition strategy that jointly evades input-level and output-level filters without model-specific fine-tuning
- Empirical demonstration that adversarial images optimized on one vision encoder broadly transfer to unseen VLMs, revealing cross-model safety vulnerabilities from shared visual representations
🛡️ Threat Analysis
Core contribution (Attention-Transfer Attack) crafts adversarial images via gradient-based optimization against a vision encoder, with perturbations that transfer cross-model to unseen VLMs — a direct adversarial input manipulation attack at inference time.