Multimodal Generative Engine Optimization: Rank Manipulation for Vision-Language Model Rankers

Vision-Language Models (VLMs) are rapidly replacing unimodal encoders in modern retrieval and recommendation systems. While their capabilities are well-documented, their robustness against adversarial manipulation in competitive ranking scenarios remains largely unexplored. In this paper, we uncover a critical vulnerability in VLM-based product search: multimodal ranking attacks. We present Multimodal Generative Engine Optimization (MGEO), a novel adversarial framework that enables a malicious actor to unfairly promote a target product by jointly optimizing imperceptible image perturbations and fluent textual suffixes. Unlike existing attacks that treat modalities in isolation, MGEO employs an alternating gradient-based optimization strategy to exploit the deep cross-modal coupling within the VLM. Extensive experiments on real-world datasets using state-of-the-art models demonstrate that our coordinated attack significantly outperforms text-only and image-only baselines. These findings reveal that multimodal synergy, typically a strength of VLMs, can be weaponized to compromise the integrity of search rankings without triggering conventional content filters.

Key Contributions

• First formulation of multimodal ranking attacks on VLM-based rerankers, modeling a realistic adversary who modifies only their own product listing under stealth constraints.
• MGEO framework integrating PGD-based imperceptible image perturbation with gradient-based soft embedding optimization for fluent adversarial text suffixes.
• Alternating optimization algorithm that exploits cross-modal coupling in VLMs, substantially outperforming text-only, image-only, and generative heuristic baselines.

🛡️ Threat Analysis

Details

Similar Papers