Expected Harm: Rethinking Safety Evaluation of (Mis)Aligned LLMs

Current evaluations of LLM safety predominantly rely on severity-based taxonomies to assess the harmfulness of malicious queries. We argue that this formulation requires re-examination as it assumes uniform risk across all malicious queries, neglecting Execution Likelihood--the conditional probability of a threat being realized given the model's response. In this work, we introduce Expected Harm, a metric that weights the severity of a jailbreak by its execution likelihood, modeled as a function of execution cost. Through empirical analysis of state-of-the-art models, we reveal a systematic Inverse Risk Calibration: models disproportionately exhibit stronger refusal behaviors for low-likelihood (high-cost) threats while remaining vulnerable to high-likelihood (low-cost) queries. We demonstrate that this miscalibration creates a structural vulnerability: by exploiting this property, we increase the attack success rate of existing jailbreaks by up to $2\times$. Finally, we trace the root cause of this failure using linear probing, which reveals that while models encode severity in their latent space to drive refusal decisions, they possess no distinguishable internal representation of execution cost, making them "blind" to this critical dimension of risk.

Key Contributions

• Expected Harm metric that weights jailbreak severity by execution likelihood (modeled as a function of execution cost) to better capture real-world risk
• Discovery of Inverse Risk Calibration — LLMs systematically over-refuse low-probability, high-cost threats while remaining vulnerable to high-probability, low-cost queries
• Linear probing analysis revealing models encode severity in latent space but possess no distinguishable internal representation of execution cost

🛡️ Threat Analysis

Details

Similar Papers