ML Security PapersShadows in the Code: Exploring the Risks and Defenses of LLM-based Multi-Agent Software Development Systems
Xiaoqing Wang 1, Keman Huang 1, Bin Liang 1, Hongyu Li 2, Xiaoyong Du 1
Published on arXiv
2511.18467
Prompt Injection
OWASP LLM Top 10 — LLM01
Excessive Agency
OWASP LLM Top 10 — LLM08
Key Finding
IMBIA achieves attack success rates of up to 93% in MU-BA and 84% in BU-MA scenarios across three multi-agent frameworks; Adv-IMBIA reduces these rates by up to 73% and 45% respectively.
IMBIA (Implicit Malicious Behavior Injection Attack)
Novel technique introduced
The rapid advancement of Large Language Model (LLM)-driven multi-agent systems has significantly streamlined software developing tasks, enabling users with little technical expertise to develop executable applications. While these systems democratize software creation through natural language requirements, they introduce significant security risks that remain largely unexplored. We identify two risky scenarios: Malicious User with Benign Agents (MU-BA) and Benign User with Malicious Agents (BU-MA). We introduce the Implicit Malicious Behavior Injection Attack (IMBIA), demonstrating how multi-agent systems can be manipulated to generate software with concealed malicious capabilities beneath seemingly benign applications, and propose Adv-IMBIA as a defense mechanism. Evaluations across ChatDev, MetaGPT, and AgentVerse frameworks reveal varying vulnerability patterns, with IMBIA achieving attack success rates of 93%, 45%, and 71% in MU-BA scenarios, and 71%, 84%, and 45% in BU-MA scenarios. Our defense mechanism reduced attack success rates significantly, particularly in the MU-BA scenario. Further analysis reveals that compromised agents in the coding and testing phases pose significantly greater security risks, while also identifying critical agents that require protection against malicious user exploitation. Our findings highlight the urgent need for robust security measures in multi-agent software development systems and provide practical guidelines for implementing targeted, resource-efficient defensive strategies.
Key Contributions
- Introduces IMBIA (Implicit Malicious Behavior Injection Attack), demonstrating how LLM multi-agent software development systems can be manipulated via malicious users (MU-BA) or compromised agents (BU-MA) to generate software with concealed malicious capabilities.
- Proposes Adv-IMBIA, a defense mechanism operating at the agent level (MU-BA) and user interface level (BU-MA), achieving up to 73% reduction in attack success rates.
- Empirically identifies that agents in coding and testing phases pose the greatest security risk in BU-MA scenarios, and provides practical guidelines for targeted, resource-efficient defense deployment across ChatDev, MetaGPT, and AgentVerse.