Mind the GAP: Text Safety Does Not Transfer to Tool-Call Safety in LLM Agents

Large language models deployed as agents increasingly interact with external systems through tool calls--actions with real-world consequences that text outputs alone do not carry. Safety evaluations, however, overwhelmingly measure text-level refusal behavior, leaving a critical question unanswered: does alignment that suppresses harmful text also suppress harmful actions? We introduce the GAP benchmark, a systematic evaluation framework that measures divergence between text-level safety and tool-call-level safety in LLM agents. We test six frontier models across six regulated domains (pharmaceutical, financial, educational, employment, legal, and infrastructure), seven jailbreak scenarios per domain, three system prompt conditions (neutral, safety-reinforced, and tool-encouraging), and two prompt variants, producing 17,420 analysis-ready datapoints. Our central finding is that text safety does not transfer to tool-call safety. Across all six models, we observe instances where the model's text output refuses a harmful request while its tool calls simultaneously execute the forbidden action--a divergence we formalize as the GAP metric. Even under safety-reinforced system prompts, 219 such cases persist across all six models. System prompt wording exerts substantial influence on tool-call behavior: TC-safe rates span 21 percentage points for the most robust model and 57 for the most prompt-sensitive, with 16 of 18 pairwise ablation comparisons remaining significant after Bonferroni correction. Runtime governance contracts reduce information leakage in all six models but produce no detectable deterrent effect on forbidden tool-call attempts themselves. These results demonstrate that text-only safety evaluations are insufficient for assessing agent behavior and that tool-call safety requires dedicated measurement and mitigation.

Key Contributions

• GAP metric formalizing the divergence where a model's text output refuses a harmful request while its tool calls simultaneously execute the forbidden action, evaluated across 17,420 datapoints on six frontier models
• First systematic three-way system prompt ablation (neutral, safety-reinforced, tool-encouraging) revealing prompt wording shifts TC-safe rates by 21–57 percentage points depending on model
• Finding that runtime governance contracts reduce information leakage (LEAK metric) across all six models but produce no detectable deterrent on forbidden tool-call attempt rates

🛡️ Threat Analysis

Details

Similar Papers