Token Buncher: Shielding LLMs from Harmful Reinforcement Learning Fine-Tuning

As large language models (LLMs) continue to grow in capability, so do the risks of harmful misuse through fine-tuning. While most prior studies assume that attackers rely on supervised fine-tuning (SFT) for such misuse, we systematically demonstrate that reinforcement learning (RL) enables adversaries to more effectively break safety alignment and facilitate more advanced harmful task assistance, under matched computational budgets. To counter this emerging threat, we propose TokenBuncher, the first effective defense specifically targeting RL-based harmful fine-tuning. TokenBuncher suppresses the foundation on which RL relies: model response entropy. By constraining entropy, RL-based fine-tuning can no longer exploit distinct reward signals to drive the model toward harmful behaviors. We realize this defense through entropy-as-reward RL and a Token Noiser mechanism designed to prevent the escalation of harmful capabilities. Extensive experiments across multiple models and RL algorithms show that TokenBuncher robustly mitigates harmful RL fine-tuning while preserving benign task performance and finetunability. Our results highlight that RL-based harmful fine-tuning poses a greater systemic risk than SFT, and that TokenBuncher provides an effective and general defense.

Key Contributions

• Systematic demonstration that RL-based harmful fine-tuning is more effective at breaking safety alignment than SFT under matched compute budgets
• TokenBuncher defense that constrains model response entropy to prevent RL from exploiting distinct reward signals for harmful behavior
• Token Noiser mechanism paired with entropy-as-reward RL to block escalation of harmful capabilities while preserving benign performance

🛡️ Threat Analysis

Details

Similar Papers