ML Security PapersBlueCodeAgent: A Blue Teaming Agent Enabled by Automated Red Teaming for CodeGen AI
Chengquan Guo 1, Yuzhou Nie 2, Chulin Xie 3, Zinan Lin 4, Wenbo Guo 2,5, Bo Li 1,3,5
Published on arXiv
2510.18131
Prompt Injection
OWASP LLM Top 10 — LLM01
Key Finding
BlueCodeAgent achieves an average 12.7% F1 score improvement across four datasets in three code-related safety tasks over base models and safety prompt-based defenses.
BlueCodeAgent
Novel technique introduced
As large language models (LLMs) are increasingly used for code generation, concerns over the security risks have grown substantially. Early research has primarily focused on red teaming, which aims to uncover and evaluate vulnerabilities and risks of CodeGen models. However, progress on the blue teaming side remains limited, as developing defense requires effective semantic understanding to differentiate the unsafe from the safe. To fill in this gap, we propose BlueCodeAgent, an end-to-end blue teaming agent enabled by automated red teaming. Our framework integrates both sides: red teaming generates diverse risky instances, while the blue teaming agent leverages these to detect previously seen and unseen risk scenarios through constitution and code analysis with agentic integration for multi-level defense. Our evaluation across three representative code-related tasks--bias instruction detection, malicious instruction detection, and vulnerable code detection--shows that BlueCodeAgent achieves significant gains over the base models and safety prompt-based defenses. In particular, for vulnerable code detection tasks, BlueCodeAgent integrates dynamic analysis to effectively reduce false positives, a challenging problem as base models tend to be over-conservative, misclassifying safe code as unsafe. Overall, BlueCodeAgent achieves an average 12.7\% F1 score improvement across four datasets in three tasks, attributed to its ability to summarize actionable constitutions that enhance context-aware risk detection. We demonstrate that the red teaming benefits the blue teaming by continuously identifying new vulnerabilities to enhance defense performance.
Key Contributions
- End-to-end BlueCodeAgent framework that integrates automated red teaming (to generate diverse risky instances) with a blue teaming agent (to detect seen and unseen risk scenarios) for CodeGen LLMs.
- Constitution and code analysis with agentic integration enabling multi-level defense across bias instruction detection, malicious instruction detection, and vulnerable code detection.
- Dynamic analysis integration to reduce false positives in vulnerable code detection, achieving an average 12.7% F1 score improvement over base models and safety prompt-based defenses.