Adversarial Humanities Benchmark: Results on Stylistic Robustness in Frontier Model Safety

The Adversarial Humanities Benchmark (AHB) evaluates whether model safety refusals survive a shift away from familiar harmful prompt forms. Starting from harmful tasks drawn from MLCommons AILuminate, the benchmark rewrites the same objectives through humanities-style transformations while preserving intent. This extends literature on Adversarial Poetry and Adversarial Tales from single jailbreak operators to a broader benchmark family of stylistic obfuscation and goal concealment. In the benchmark results reported here, the original attacks record 3.84% attack success rate (ASR), while transformed methods range from 36.8% to 65.0%, yielding 55.75% overall ASR across 31 frontier models. Under a European Union AI Act Code-of-Practice-inspired systemic-risk lens, Chemical, biological, radiological and nuclear (CBRN) is the highest bucket. Taken together, this lack of stylistic robustness suggests that current safety techniques suffer from weak generalization: deep understanding of 'non-maleficence' remains a central unresolved problem in frontier model safety.

Key Contributions

• Adversarial Humanities Benchmark (AHB) - automated framework for generating stylistically obfuscated jailbreak prompts using literary/philosophical transformations
• Evaluation across 31 frontier models showing 55.75% overall ASR (vs 3.84% baseline), demonstrating weak generalization of safety guardrails
• Extends Adversarial Poetry and Adversarial Tales to a broader benchmark family measuring stylistic robustness gap in LLM safety

🛡️ Threat Analysis

Details

Similar Papers