ML Security PapersThe Missing Half: Unveiling Training-time Implicit Safety Risks Beyond Deployment
Zhexin Zhang 1, Yida Lu 1, Junfeng Fang 2, Junxiao Yang 1, Shiyao Cui 1, Hao Zhou 3, Fandong Meng 3, Jie Zhou 3, Hongning Wang 1, Minlie Huang 1, Tat-Seng Chua 2
Published on arXiv
2602.04196
Model Skewing
OWASP ML Top 10 — ML08
Excessive Agency
OWASP LLM Top 10 — LLM08
Key Finding
Llama-3.1-8B-Instruct exhibits implicit risky behaviors in 74.4% of training runs when supplied only with background information, revealing a pervasive and underexplored training-time safety threat
Safety risks of AI models have been widely studied at deployment time, such as jailbreak attacks that elicit harmful outputs. In contrast, safety risks emerging during training remain largely unexplored. Beyond explicit reward hacking that directly manipulates explicit reward functions in reinforcement learning, we study implicit training-time safety risks: harmful behaviors driven by a model's internal incentives and contextual background information. For example, during code-based reinforcement learning, a model may covertly manipulate logged accuracy for self-preservation. We present the first systematic study of this problem, introducing a taxonomy with five risk levels, ten fine-grained risk categories, and three incentive types. Extensive experiments reveal the prevalence and severity of these risks: notably, Llama-3.1-8B-Instruct exhibits risky behaviors in 74.4% of training runs when provided only with background information. We further analyze factors influencing these behaviors and demonstrate that implicit training-time risks also arise in multi-agent training settings. Our results identify an overlooked yet urgent safety challenge in training.
Key Contributions
- First systematic taxonomy of training-time implicit safety risks with 5 risk levels, 10 fine-grained categories, and 3 incentive types
- Extensive experiments demonstrating Llama-3.1-8B-Instruct exhibits risky behaviors in 74.4% of RL training runs given only background information
- Analysis of implicit safety risks in multi-agent training settings, expanding the threat surface beyond single-model RL
🛡️ Threat Analysis
The paper studies reward hacking over time in RL training systems — models developing harmful behaviors through internal incentives and feedback loops during training, which is the canonical ML08 threat (reward hacking in RL systems).