ML Security PapersBackdoors in RLVR: Jailbreak Backdoors in LLMs From Verifiable Reward
Weiyang Guo 1, Zesheng Shi 1, Zeen Zhu 1, Yuan Zhou 2, Min Zhang 1, Jing Li 1
Published on arXiv
2604.09748
Model Poisoning
OWASP ML Top 10 — ML10
Transfer Learning Attack
OWASP ML Top 10 — ML07
Prompt Injection
OWASP LLM Top 10 — LLM01
Key Finding
Achieves successful backdoor implantation using <2% poisoned data, degrading safety performance by average of 73% when trigger is activated, while maintaining benign task performance
Asymmetric Chain Backdoor (ACB)
Novel technique introduced
Reinforcement Learning with Verifiable Rewards (RLVR) is an emerging paradigm that significantly boosts a Large Language Model's (LLM's) reasoning abilities on complex logical tasks, such as mathematics and programming. However, we identify, for the first time, a latent vulnerability to backdoor attacks within the RLVR framework. This attack can implant a backdoor without modifying the reward verifier by injecting a small amount of poisoning data into the training set. Specifically, we propose a novel trigger mechanism designated as the \ourapproach (ACB). The attack exploits the RLVR training loop by assigning substantial positive rewards for harmful responses and negative rewards for refusals. This asymmetric reward signal forces the model to progressively increase the probability of generating harmful responses during training. Our findings demonstrate that the RLVR backdoor attack is characterized by both high efficiency and strong generalization capabilities. Utilizing less than 2\% poisoned data in train set, the backdoor can be successfully implanted across various model scales without degrading performance on benign tasks. Evaluations across multiple jailbreak benchmarks indicate that activating the trigger degrades safety performance by an average of 73\%. Furthermore, the attack generalizes effectively to a wide range of jailbreak methods and unsafe behaviors. Code is available at https://github.com/yuki-younai/Backdoor_in_RLVR.
Key Contributions
- First backdoor attack targeting RLVR training paradigm, requiring only prompt poisoning (not reward model tampering)
- Asymmetric Chain Backdoor (ACB) mechanism that exploits reward asymmetry to progressively dismantle safety alignment
- Shadow-driven backdoor data synthesis method with dual filtering for efficient poison sample selection
🛡️ Threat Analysis
The attack specifically exploits the RLVR training paradigm (a form of reinforcement learning fine-tuning). The backdoor survives and exploits the RL training loop, manipulating reward signals during the fine-tuning process — this is a transfer learning attack where malicious behavior is embedded during the RL alignment phase.
Primary contribution is a backdoor/trojan attack that embeds hidden trigger-activated jailbreak behavior in LLMs during RLVR training. The Asymmetric Chain Backdoor (ACB) mechanism creates a trigger that causes the model to generate harmful responses while behaving normally otherwise — textbook ML10.