ML Security PapersRedCodeAgent: Automatic Red-teaming Agent against Diverse Code Agents
Chengquan Guo 1, Chulin Xie 2,3, Yu Yang 4,5, Zhaorun Chen 1, Zinan Lin 1, Xander Davies 6, Yarin Gal 6,7, Dawn Song 5, Bo Li
Published on arXiv
2510.02609
Prompt Injection
OWASP LLM Top 10 — LLM01
Key Finding
RedCodeAgent achieves higher attack success rates and lower rejection rates than existing red-teaming methods across multiple state-of-the-art code agents, and exposes previously unidentified vulnerabilities in Cursor and Codeium
RedCodeAgent
Novel technique introduced
Code agents have gained widespread adoption due to their strong code generation capabilities and integration with code interpreters, enabling dynamic execution, debugging, and interactive programming capabilities. While these advancements have streamlined complex workflows, they have also introduced critical safety and security risks. Current static safety benchmarks and red-teaming tools are inadequate for identifying emerging real-world risky scenarios, as they fail to cover certain boundary conditions, such as the combined effects of different jailbreak tools. In this work, we propose RedCodeAgent, the first automated red-teaming agent designed to systematically uncover vulnerabilities in diverse code agents. With an adaptive memory module, RedCodeAgent can leverage existing jailbreak knowledge, dynamically select the most effective red-teaming tools and tool combinations in a tailored toolbox for a given input query, thus identifying vulnerabilities that might otherwise be overlooked. For reliable evaluation, we develop simulated sandbox environments to additionally evaluate the execution results of code agents, mitigating potential biases of LLM-based judges that only rely on static code. Through extensive evaluations across multiple state-of-the-art code agents, diverse risky scenarios, and various programming languages, RedCodeAgent consistently outperforms existing red-teaming methods, achieving higher attack success rates and lower rejection rates with high efficiency. We further validate RedCodeAgent on real-world code assistants, e.g., Cursor and Codeium, exposing previously unidentified security risks. By automating and optimizing red-teaming processes, RedCodeAgent enables scalable, adaptive, and effective safety assessments of code agents.
Key Contributions
- RedCodeAgent: first automated red-teaming agent with an adaptive memory module that dynamically selects and combines jailbreak tools from a curated toolbox to maximize attack success against diverse code agents
- Simulated sandbox environments for reliable evaluation of code agent outputs, mitigating LLM-judge bias when assessing executed (not just static) code
- Empirical validation on real-world code assistants (Cursor, Codeium), uncovering previously unknown security vulnerabilities across multiple programming languages and risky scenarios