defense arXiv Jan 8, 2026 · 12w ago
Zhiyuan Chang, Mingyang Li, Yuekai Huang et al. · State Key Laboratory of Complex System Modeling and Simulation Technology · Institute of Software Chinese Academy of Sciences +3 more
Defends LLMs against prompt injection via diverse synthetic training data and instruction-level chain-of-thought fine-tuning
Prompt Injection nlp
Large language model (LLM)-integrated applications have become increasingly prevalent, yet face critical security vulnerabilities from prompt injection (PI) attacks. Defending against PI attacks faces two major issues: malicious instructions can be injected through diverse vectors, and injected instructions often lack clear semantic boundaries from the surrounding context, making them difficult to identify. To address these issues, we propose InstruCoT, a model enhancement method for PI defense that synthesizes diverse training data and employs instruction-level chain-of-thought fine-tuning, enabling LLMs to effectively identify and reject malicious instructions regardless of their source or position in the context. We evaluate InstruCoT across three critical dimensions: Behavior Deviation, Privacy Leakage, and Harmful Output. Experimental results across four LLMs demonstrate that InstruCoT significantly outperforms baselines in all dimensions while maintaining utility performance without degradation
llm transformer State Key Laboratory of Complex System Modeling and Simulation Technology · Institute of Software Chinese Academy of Sciences · University of Chinese Academy of Sciences +2 more
ML Security Papers