Gradient Surgery for Safe LLM Fine-Tuning

Fine-tuning-as-a-Service introduces a critical vulnerability where a few malicious examples mixed into the user's fine-tuning dataset can compromise the safety alignment of Large Language Models (LLMs). While a recognized paradigm frames safe fine-tuning as a multi-objective optimization problem balancing user task performance with safety alignment, we find existing solutions are critically sensitive to the harmful ratio, with defenses degrading sharply as harmful ratio increases. We diagnose that this failure stems from conflicting gradients, where the user-task update directly undermines the safety objective. To resolve this, we propose SafeGrad, a novel method that employs gradient surgery. When a conflict is detected, SafeGrad nullifies the harmful component of the user-task gradient by projecting it onto the orthogonal plane of the alignment gradient, allowing the model to learn the user's task without sacrificing safety. To further enhance robustness and data efficiency, we employ a KL-divergence alignment loss that learns the rich, distributional safety profile of the well-aligned foundation model. Extensive experiments show that SafeGrad provides state-of-the-art defense across various LLMs and datasets, maintaining robust safety even at high harmful ratios without compromising task fidelity.

Key Contributions

• Diagnoses gradient conflict — where user-task updates directly oppose alignment gradients — as the root cause of safety degradation under harmful fine-tuning attacks
• Proposes SafeGrad, which projects user-task gradients onto the orthogonal complement of the alignment gradient to nullify their harmful component while preserving task learning
• Employs a KL-divergence alignment loss to capture the full distributional safety profile of the well-aligned foundation model, improving robustness especially at high harmful data ratios

🛡️ Threat Analysis

Details

Similar Papers