ML Security PapersBadBlocks: Lightweight and Stealthy Backdoor Threat in Text-to-Image Diffusion Models
Yu Pan, Jiahao Chen, Wenjie Wang et al. · ShanghaiTech University · Shanghai Polytechnic University +1 more
Lightweight backdoor attack on text-to-image diffusion models targeting only select UNet blocks, slashing GPU cost 5x while evading attention-based defenses
Diffusion models have recently achieved remarkable success in image generation, yet growing evidence shows their vulnerability to backdoor attacks, where adversaries implant covert triggers to manipulate outputs. While existing defenses can detect many such attacks via visual inspection and neural network-based analysis, we identify a more lightweight and stealthy threat, termed BadBlocks. BadBlocks selectively contaminates specific blocks within the UNet architecture while preserving the normal behavior of the remaining components. Compared with prior methods, it requires only about 30% of the computation and 20% of the GPU time, yet achieves high attack success rates with minimal perceptual degradation. Extensive experiments demonstrate that BadBlocks can effectively evade state-of-the-art defenses, particularly attention-based detection frameworks. Ablation studies further reveal that effective backdoor injection does not require fine-tuning the entire network and highlight the critical role of certain layers in backdoor mapping. Overall, BadBlocks substantially lowers the barrier for backdooring large-scale diffusion models, even on consumer-grade GPUs.