Unifying Adversarial Robustness and Training Across Text Scoring Models

Research on adversarial robustness in language models is currently fragmented across applications and attacks, obscuring shared vulnerabilities. In this work, we propose unifying the study of adversarial robustness in text scoring models spanning dense retrievers, rerankers, and reward models. This motivates adapting both attacks and adversarial training methods across model roles. Unlike open-ended generation, text scoring failures are directly testable: an attack succeeds when an irrelevant or rejected text outscores a relevant or chosen one. Using this principled lens of text scoring, we demonstrate that current adversarial training formulations for language models are often short-sighted, failing to effectively generalize across attacks. To address this, we introduce multiple adversarial training methods for text scoring models and show that combining complementary training methods can yield strong robustness while also improving task effectiveness. We also highlight the practical value of our approach for RLHF, showing that our adversarially trained reward models mitigate reward hacking and support the training of better-aligned LLMs. We provide our code and models for further study.

Key Contributions

• Unifying adversarial robustness study across dense retrievers, rerankers, and reward models under the text scoring lens with crisp, testable failure conditions
• Introduction of adversarial training against content injection attacks — a previously unaddressed threat in this setting — alongside PGD, HotFlip, and rudimentary adversarial training
• Demonstration that combining complementary adversarial training methods improves both robustness and task effectiveness, with adversarially trained reward models mitigating reward hacking in RLHF

🛡️ Threat Analysis

Details

Similar Papers