A methodological analysis of prompt perturbations and their effect on attack success rates

This work aims to investigate how different Large Language Models (LLMs) alignment methods affect the models' responses to prompt attacks. We selected open source models based on the most common alignment methods, namely, Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Reinforcement Learning with Human Feedback (RLHF). We conducted a systematic analysis using statistical methods to verify how sensitive the Attack Success Rate (ASR) is when we apply variations to prompts designed to elicit inappropriate content from LLMs. Our results show that even small prompt modifications can significantly change the Attack Success Rate (ASR) according to the statistical tests we run, making the models more or less susceptible to types of attack. Critically, our results demonstrate that running existing 'attack benchmarks' alone may not be sufficient to elicit all possible vulnerabilities of both models and alignment methods. This paper thus contributes to ongoing efforts on model attack evaluation by means of systematic and statistically-based analyses of the different alignment methods and how sensitive their ASR is to prompt variation.

Key Contributions

• Systematic statistical analysis (significance testing) of ASR sensitivity to prompt perturbations across SFT, DPO, and RLHF alignment methods
• Demonstrates that even small prompt modifications can produce statistically significant changes in ASR, challenging the reliability of single-prompt attack benchmarks
• Highlights that existing attack benchmark protocols (AdvBench, HarmBench) may underestimate the range of vulnerabilities in aligned LLMs

🛡️ Threat Analysis

Details

Similar Papers