RAG Makes Guardrails Unsafe? Investigating Robustness of Guardrails under RAG-style Contexts

With the increasing adoption of large language models (LLMs), ensuring the safety of LLM systems has become a pressing concern. External LLM-based guardrail models have emerged as a popular solution to screen unsafe inputs and outputs, but they are themselves fine-tuned or prompt-engineered LLMs that are vulnerable to data distribution shifts. In this paper, taking Retrieval Augmentation Generation (RAG) as a case study, we investigated how robust LLM-based guardrails are against additional information embedded in the context. Through a systematic evaluation of 3 Llama Guards and 2 GPT-oss models, we confirmed that inserting benign documents into the guardrail context alters the judgments of input and output guardrails in around 11% and 8% of cases, making them unreliable. We separately analyzed the effect of each component in the augmented context: retrieved documents, user query, and LLM-generated response. The two mitigation methods we tested only bring minor improvements. These results expose a context-robustness gap in current guardrails and motivate training and evaluation protocols that are robust to retrieval and query composition.

Key Contributions

• Systematic evaluation showing benign RAG-retrieved documents alter LLM guardrail safety judgments in ~11% (input) and ~8% (output) of cases across 5 models (3 Llama Guard variants, 2 GPT-oss)
• Ablation analysis isolating the effect of individual RAG context components (retrieved documents, user query, LLM-generated response) on guardrail judgment shifts
• Evaluation of two mitigation strategies, finding only marginal improvements and motivating robustness-aware guardrail training and evaluation protocols

🛡️ Threat Analysis

Details

Similar Papers