ML Security PapersSafeReview: Defending LLM-based Review Systems Against Adversarial Hidden Prompts
Yuan Xin 1, Yixuan Weng 2, Minjun Zhu 2, Ying Ling 3, Chengwei Qin 4, Michael Hahn 5, Michael Backes 1, Yue Zhang 2, Linyi Yang 3
Published on arXiv
2604.26506
Prompt Injection
OWASP LLM Top 10 — LLM01
Key Finding
Significantly reduces acceptance rate of adversarially manipulated papers under adaptive GRPO attacks while achieving highest Spearman correlation with ground-truth scores among all defense methods
SafeReview
Novel technique introduced
As Large Language Models (LLMs) are increasingly integrated into academic peer review, their vulnerability to adversarial prompts -- adversarial instructions embedded in submissions to manipulate outcomes -- emerges as a critical threat to scholarly integrity. To counter this, we propose a novel adversarial framework where a Generator model, trained to create sophisticated attack prompts, is jointly optimized with a Defender model tasked with their detection. This system is trained using a loss function inspired by Information Retrieval Generative Adversarial Networks, which fosters a dynamic co-evolution between the two models, forcing the Defender to develop robust capabilities against continuously improving attack strategies. The resulting framework demonstrates significantly enhanced resilience to novel and evolving threats compared to static defenses, thereby establishing a critical foundation for securing the integrity of peer review.
Key Contributions
- First co-evolutionary adversarial training framework (Generator vs Defender) for defending LLM-based peer review against hidden prompt injection attacks
- Stable training pipeline combining GRPO-based attack generation with DPO-based defense, tailored for long-form academic documents
- Demonstrates superior robustness and ranking preservation compared to static defenses while maintaining fairness on benign submissions