ML Security PapersSelective Steering: Norm-Preserving Control Through Discriminative Layer Selection
Quy-Anh Dang 1,2, Chris Ngo 1,2
Published on arXiv
2601.19375
Prompt Injection
OWASP LLM Top 10 — LLM01
Key Finding
Selective Steering achieves 5.5x higher attack success rates than prior activation steering methods across nine models while maintaining zero perplexity violations and approximately 100% capability retention on standard benchmarks.
Selective Steering
Novel technique introduced
Despite significant progress in alignment, large language models (LLMs) remain vulnerable to adversarial attacks that elicit harmful behaviors. Activation steering techniques offer a promising inference-time intervention approach, but existing methods suffer from critical limitations: activation addition requires careful coefficient tuning and is sensitive to layer-specific norm variations, while directional ablation provides only binary control. Recent work on Angular Steering introduces continuous control via rotation in a 2D subspace, but its practical implementation violates norm preservation, causing distribution shift and generation collapse, particularly in models below 7B parameters. We propose Selective Steering, which addresses these limitations through two key innovations: (1) a mathematically rigorous norm-preserving rotation formulation that maintains activation distribution integrity, and (2) discriminative layer selection that applies steering only where feature representations exhibit opposite-signed class alignment. Experiments across nine models demonstrate that Selective Steering achieves 5.5x higher attack success rates than prior methods while maintaining zero perplexity violations and approximately 100\% capability retention on standard benchmarks. Our approach provides a principled, efficient framework for controllable and stable LLM behavior modification. Code: https://github.com/knoveleng/steering
Key Contributions
- Mathematically rigorous norm-preserving rotation formulation for activation steering that avoids distribution shift and generation collapse seen in prior Angular Steering methods
- Discriminative layer selection strategy that applies steering only to layers where feature representations exhibit opposite-signed class alignment, improving efficiency and effectiveness
- 5.5x improvement in attack success rate over prior activation steering methods across nine LLMs while maintaining zero perplexity violations and ~100% capability retention