ML Security PapersClaudini: Autoresearch Discovers State-of-the-Art Adversarial Attack Algorithms for LLMs
Alexander Panfilov 1,2,3,4, Peter Romov 5, Igor Shilov 2,3,4, Yves-Alexandre de Montjoye 5, Jonas Geiping 2,3,4, Maksym Andriushchenko 5
Published on arXiv
2603.24511
Input Manipulation Attack
OWASP ML Top 10 — ML01
Prompt Injection
OWASP LLM Top 10 — LLM01
Key Finding
Achieves 100% attack success rate against Meta-SecAlign-70B (vs 56% for best baseline GCG) and 40% ASR on CBRN queries against GPT-OSS-Safeguard-20B (vs ≤10% for existing methods)
Claudini
Novel technique introduced
LLM agents like Claude Code can not only write code but also be used for autonomous AI research and engineering \citep{rank2026posttrainbench, novikov2025alphaevolve}. We show that an \emph{autoresearch}-style pipeline \citep{karpathy2026autoresearch} powered by Claude Code discovers novel white-box adversarial attack \textit{algorithms} that \textbf{significantly outperform all existing (30+) methods} in jailbreaking and prompt injection evaluations. Starting from existing attack implementations, such as GCG~\citep{zou2023universal}, the agent iterates to produce new algorithms achieving up to 40\% attack success rate on CBRN queries against GPT-OSS-Safeguard-20B, compared to $\leq$10\% for existing algorithms (\Cref{fig:teaser}, left). The discovered algorithms generalize: attacks optimized on surrogate models transfer directly to held-out models, achieving \textbf{100\% ASR against Meta-SecAlign-70B} \citep{chen2025secalign} versus 56\% for the best baseline (\Cref{fig:teaser}, middle). Extending the findings of~\cite{carlini2025autoadvexbench}, our results are an early demonstration that incremental safety and security research can be automated using LLM agents. White-box adversarial red-teaming is particularly well-suited for this: existing methods provide strong starting points, and the optimization objective yields dense, quantitative feedback. We release all discovered attacks alongside baseline implementations and evaluation code at https://github.com/romovpa/claudini.
Key Contributions
- Autoresearch pipeline using LLM agents (Claude Code) to autonomously discover novel adversarial attack algorithms
- Discovered attacks achieve 40% ASR on CBRN queries (vs ≤10% for baselines) and 100% transfer ASR on Meta-SecAlign-70B (vs 56% for best baseline)
- Demonstrates that white-box adversarial red-teaming research can be automated through iterative LLM agent refinement of existing methods like GCG
🛡️ Threat Analysis
Develops gradient-based adversarial suffix optimization attacks (white-box) that cause misclassification/harmful outputs at inference time by crafting adversarial token sequences.