ML Security PapersWhen Surfaces Lie: Exploiting Wrinkle-Induced Attention Shift to Attack Vision-Language Models
Chengyin Hu , Xuemeng Sun , Jiajun Han , Qike Zhang , Xiang Chen , Xin Wang , Yiwei Wei , Jiahua Long
Published on arXiv
2603.27759
Input Manipulation Attack
OWASP ML Top 10 — ML01
Prompt Injection
OWASP LLM Top 10 — LLM01
Key Finding
Wrinkle-based perturbations significantly degrade performance across multiple SOTA VLMs on classification, captioning, and VQA tasks, outperforming baseline attacks while maintaining visual naturalness
Wrinkle-Induced Structural Attack
Novel technique introduced
Visual-Language Models (VLMs) have demonstrated exceptional cross-modal understanding across various tasks, including zero-shot classification, image captioning, and visual question answering. However, their robustness to physically plausible non-rigid deformations-such as wrinkles on flexible surfaces-remains poorly understood. In this work, we propose a parametric structural perturbation method inspired by the mechanics of three-dimensional fabric wrinkles. Specifically, our method generates photorealistic non-rigid perturbations by constructing multi-scale wrinkle fields and integrating displacement field distortion with surface-consistent appearance variations. To achieve an optimal balance between visual naturalness and adversarial effectiveness, we design a hierarchical fitness function in a low-dimensional parameter space and employ an optimization-based search strategy. We evaluate our approach using a two-stage framework: perturbations are first optimized on a zero-shot classification proxy task and subsequently assessed for transferability on generative tasks. Experimental results demonstrate that our method significantly degrades the performance of various state-of-the-art VLMs, consistently outperforming baselines in both image captioning and visual question-answering tasks.
Key Contributions
- Parametric wrinkle-based structural perturbation method using multi-scale wrinkle fields, displacement warping, and surface-consistent appearance variation
- Hierarchical fitness function with genetic algorithm optimization in low-dimensional parameter space for physically plausible adversarial examples
- Two-stage attack framework: optimize on zero-shot classification proxy task, then transfer to generative tasks (image captioning, VQA) across multiple SOTA VLMs
🛡️ Threat Analysis
Proposes a novel input manipulation attack using wrinkle-like structural perturbations to cause misclassification and degraded performance in VLMs at inference time. The attack uses optimization-based search (genetic algorithm) to find effective perturbation parameters that manipulate visual inputs, causing performance degradation across zero-shot classification, image captioning, and visual question answering tasks.