A Low-Rank Defense Method for Adversarial Attack on Diffusion Models

Recently, adversarial attacks for diffusion models as well as their fine-tuning process have been developed rapidly. To prevent the abuse of these attack algorithms from affecting the practical application of diffusion models, it is critical to develop corresponding defensive strategies. In this work, we propose an efficient defensive strategy, named Low-Rank Defense (LoRD), to defend the adversarial attack on Latent Diffusion Models (LDMs). LoRD introduces the merging idea and a balance parameter, combined with the low-rank adaptation (LoRA) modules, to detect and defend the adversarial samples. Based on LoRD, we build up a defense pipeline that applies the learned LoRD modules to help diffusion models defend against attack algorithms. Our method ensures that the LDM fine-tuned on both adversarial and clean samples can still generate high-quality images. To demonstrate the effectiveness of our approach, we conduct extensive experiments on facial and landscape images, and our method shows significantly better defense performance compared to the baseline methods.

Key Contributions

• Proposes LoRD (Low-Rank Defense), a LoRA-based module using an MLP-derived balance parameter to detect and neutralize adversarially perturbed images during diffusion model fine-tuning
• Introduces a two-stage defense pipeline: Stage-1 learns LoRD modules via adversarial training ideas, Stage-2 merges LoRD weights and fine-tunes the LDM on both adversarial and clean samples
• Demonstrates significantly better defense performance than baseline methods on facial and landscape image datasets against ACE/ACE+ attacks

🛡️ Threat Analysis

Details

Similar Papers