ML Security PapersAmnesia: Adversarial Semantic Layer Specific Activation Steering in Large Language Models
Ali Raza 1,2, Gurang Gupta 1,2, Nikolay Matyunin 1,2, Jibesh Patra 1,2
Published on arXiv
2603.10080
Prompt Injection
OWASP LLM Top 10 — LLM01
Key Finding
Amnesia bypasses RLHF-based safety mechanisms in open-weight LLMs by manipulating internal transformer activations at inference time, requiring no fine-tuning or gradient-based optimization, and successfully induces harmful content generation across benchmark datasets
Amnesia
Novel technique introduced
Warning: This article includes red-teaming experiments, which contain examples of compromised LLM responses that may be offensive or upsetting. Large Language Models (LLMs) have the potential to create harmful content, such as generating sophisticated phishing emails and assisting in writing code of harmful computer viruses. Thus, it is crucial to ensure their safe and responsible response generation. To reduce the risk of generating harmful or irresponsible content, researchers have developed techniques such as reinforcement learning with human feedback to align LLM's outputs with human values and preferences. However, it is still undetermined whether such measures are sufficient to prevent LLMs from generating interesting responses. In this study, we propose Amnesia, a lightweight activation-space adversarial attack that manipulates internal transformer states to bypass existing safety mechanisms in open-weight LLMs. Through experimental analysis on state-of-the-art, open-weight LLMs, we demonstrate that our attack effectively circumvents existing safeguards, enabling the generation of harmful content without the need for any fine-tuning or additional training. Our experiments on benchmark datasets show that the proposed attack can induce various antisocial behaviors in LLMs. These findings highlight the urgent need for more robust security measures in open-weight LLMs and underscore the importance of continued research to prevent their potential misuse.
Key Contributions
- Amnesia: a lightweight, training-free activation-space attack that manipulates layer-specific internal transformer states to bypass RLHF safety mechanisms in open-weight LLMs
- Demonstrates that safety alignment can be circumvented without fine-tuning, model editing, or cumbersome prompt optimization by directly intervening in the activation space
- Empirical evaluation on state-of-the-art open-weight LLMs showing induction of various antisocial behaviors on benchmark jailbreak datasets