ML Security PapersARGUS: Defending Against Multimodal Indirect Prompt Injection via Steering Instruction-Following Behavior
Weikai Lu 1, Ziqian Zeng 1, Kehuan Zhang 1, Haoran Li 2, Huiping Zhuang 1, Ruidong Wang 3, Cen Chen 1, Hao Peng 4
Published on arXiv
2512.05745
Prompt Injection
OWASP LLM Top 10 — LLM01
Key Finding
ARGUS achieves robust defense against multimodal IPI attacks across image, video, and audio modalities while maximally preserving MLLM utility through decoupled direction search and adaptive steering strength
ARGUS
Novel technique introduced
Multimodal Large Language Models (MLLMs) are increasingly vulnerable to multimodal Indirect Prompt Injection (IPI) attacks, which embed malicious instructions in images, videos, or audio to hijack model behavior. Existing defenses, designed primarily for text-only LLMs, are unsuitable for countering these multimodal threats, as they are easily bypassed, modality-dependent, or generalize poorly. Inspired by activation steering researches, we hypothesize that a robust, general defense independent of modality can be achieved by steering the model's behavior in the representation space. Through extensive experiments, we discover that the instruction-following behavior of MLLMs is encoded in a subspace. Steering along directions within this subspace can enforce adherence to user instructions, forming the basis of a defense. However, we also found that a naive defense direction could be coupled with a utility-degrading direction, and excessive intervention strength harms model performance. To address this, we propose ARGUS, which searches for an optimal defense direction within the safety subspace that decouples from the utility degradation direction, further combining adaptive strength steering to achieve a better safety-utility trade-off. ARGUS also introduces lightweight injection detection stage to activate the defense on-demand, and a post-filtering stage to verify defense success. Experimental results show that ARGUS can achieve robust defense against multimodal IPI while maximally preserving the MLLM's utility.
Key Contributions
- Discovers that MLLM instruction-following behavior is encoded in a subspace of the representation space, enabling modality-agnostic defense via activation steering
- Proposes ARGUS, which finds an optimal defense direction that decouples safety from utility degradation and applies adaptive strength steering for a better safety-utility trade-off
- Introduces a lightweight injection detection stage and a post-filtering verification stage to activate defense on-demand and confirm its success