ML Security PapersJailbreaking LLMs Without Gradients or Priors: Effective and Transferable Attacks
Zhakshylyk Nurlanov 1,2, Frank R. Schmidt 2, Florian Bernard 1
Published on arXiv
2601.03420
Input Manipulation Attack
OWASP ML Top 10 — ML01
Prompt Injection
OWASP LLM Top 10 — LLM01
Key Finding
Achieves 90.2% attack success rate on Llama2 and near 100% on multiple open-source models, with strong black-box transferability to GPT and Gemini via cross-tokenizer ensemble optimization.
RAILS (RAndom Iterative Local Search)
Novel technique introduced
As Large Language Models (LLMs) are increasingly deployed in safety-critical domains, rigorously evaluating their robustness against adversarial jailbreaks is essential. However, current safety evaluations often overestimate robustness because existing automated attacks are limited by restrictive assumptions. They typically rely on handcrafted priors or require white-box access for gradient propagation. We challenge these constraints by demonstrating that token-level iterative optimization can succeed without gradients or priors. We introduce RAILS (RAndom Iterative Local Search), a framework that operates solely on model logits. RAILS matches the effectiveness of gradient-based methods through two key innovations: a novel auto-regressive loss that enforces exact prefix matching, and a history-based selection strategy that bridges the gap between the proxy optimization objective and the true attack success rate. Crucially, by eliminating gradient dependency, RAILS enables cross-tokenizer ensemble attacks. This allows for the discovery of shared adversarial patterns that generalize across disjoint vocabularies, significantly enhancing transferability to closed-source systems. Empirically, RAILS achieves near 100% success rates on multiple open-source models and high black-box attack transferability to closed-source systems like GPT and Gemini.
Key Contributions
- RAILS: a gradient-free, prior-free iterative local search framework for adversarial suffix optimization that operates solely on model logits, matching gradient-based methods like GCG without white-box access
- A novel auto-regressive loss enforcing exact prefix matching and a history-based candidate selection strategy that bridges the gap between proxy objective and true attack success rate
- Cross-tokenizer ensemble attacks that discover shared adversarial patterns generalizing across disjoint vocabularies, enabling high transferability to closed-source models like GPT and Gemini
🛡️ Threat Analysis
RAILS performs token-level iterative optimization of adversarial suffixes in discrete token space — the same threat class as GCG, but gradient-free and logit-based. This is adversarial suffix optimization (not natural-language prompt manipulation), squarely within ML01.