Training a General Purpose Automated Red Teaming Model

Automated methods for red teaming LLMs are an important tool to identify LLM vulnerabilities that may not be covered in static benchmarks, allowing for more thorough probing. They can also adapt to each specific LLM to discover weaknesses unique to it. Most current automated red teaming methods are intended for tackling safety and content moderation. Thus, they make use of content safety models as evaluators and optimize for circumventing them, and as such, have not been tested with other adversarial intents not typically captured by these. We propose a pipeline for training a red teaming model that can generalize to arbitrary adversarial goals, including objectives it has not been directly trained on, and that does not depend on the existence of a pre-existing evaluator available at training time. We demonstrate that finetuning small models, such as Qwen3-8B, using this pipeline results in a substantial improvement in their ability to generate attacks for both in and out of domain adversarial goals.

Key Contributions

• Pipeline for training red teaming models that generalize to arbitrary adversarial goals, including objectives not seen during training
• Method that does not require pre-existing evaluators at training time, using LLM-generated rubrics for reward calculation
• Demonstrates substantial improvement in attack generation for both in-domain and out-of-domain adversarial goals by fine-tuning smaller models like Qwen3-8B

🛡️ Threat Analysis

Details

Similar Papers