ML Security PapersTrinityGuard: A Unified Framework for Safeguarding Multi-Agent Systems
Kai Wang , Biaojie Zeng , Zeming Wei , Chang Jin , Hefeng Zhou , Xiangtian Li , Chao Yang , Jingjing Qu , Xingcheng Xu , Xia Hu
Published on arXiv
2603.15408
Prompt Injection
OWASP LLM Top 10 — LLM01
Excessive Agency
OWASP LLM Top 10 — LLM08
Key Finding
Presents comprehensive framework covering 20 distinct risk types with both pre-development evaluation and runtime monitoring capabilities for LLM-based multi-agent systems
TrinityGuard
Novel technique introduced
With the rapid development of LLM-based multi-agent systems (MAS), their significant safety and security concerns have emerged, which introduce novel risks going beyond single agents or LLMs. Despite attempts to address these issues, the existing literature lacks a cohesive safeguarding system specialized for MAS risks. In this work, we introduce TrinityGuard, a comprehensive safety evaluation and monitoring framework for LLM-based MAS, grounded in the OWASP standards. Specifically, TrinityGuard encompasses a three-tier fine-grained risk taxonomy that identifies 20 risk types, covering single-agent vulnerabilities, inter-agent communication threats, and system-level emergent hazards. Designed for scalability across various MAS structures and platforms, TrinityGuard is organized in a trinity manner, involving an MAS abstraction layer that can be adapted to any MAS structures, an evaluation layer containing risk-specific test modules, alongside runtime monitor agents coordinated by a unified LLM Judge Factory. During Evaluation, TrinityGuard executes curated attack probes to generate detailed vulnerability reports for each risk type, where monitor agents analyze structured execution traces and issue real-time alerts, enabling both pre-development evaluation and runtime monitoring. We further formalize these safety metrics and present detailed case studies across various representative MAS examples, showcasing the versatility and reliability of TrinityGuard. Overall, TrinityGuard acts as a comprehensive framework for evaluating and monitoring various risks in MAS, paving the way for further research into their safety and security.
Key Contributions
- Three-tier risk taxonomy identifying 20 risk types across single-agent, inter-agent, and system-level threats in MAS
- Unified evaluation and runtime monitoring framework adaptable to various MAS structures and platforms
- OWASP-grounded safety metrics with curated attack probes and real-time alert system via monitor agents