ML Security PapersProjected Gradient Unlearning for Text-to-Image Diffusion Models: Defending Against Concept Revival Attacks
Aljalila Aladawi 1, Mohammed Talha Alam 1, Fakhri Karray 1,2
Published on arXiv
2604.21041
Model Inversion Attack
OWASP ML Top 10 — ML03
Key Finding
Eliminates concept revival for style concepts and substantially delays it for object concepts, running in 6 minutes versus ~2 hours for Meta-Unlearning
Projected Gradient Unlearning (PGU)
Novel technique introduced
Machine unlearning for text-to-image diffusion models aims to selectively remove undesirable concepts from pre-trained models without costly retraining. Current unlearning methods share a common weakness: erased concepts return when the model is fine-tuned on downstream data, even when that data is entirely unrelated. We adapt Projected Gradient Unlearning (PGU) from classification to the diffusion domain as a post-hoc hardening step. By constructing a Core Gradient Space (CGS) from the retain concept activations and projecting gradient updates into its orthogonal complement, PGU ensures that subsequent fine-tuning cannot undo the achieved erasure. Applied on top of existing methods (ESD, UCE, Receler), the approach eliminates revival for style concepts and substantially delays it for object concepts, running in roughly 6 minutes versus the ~2 hours required by Meta-Unlearning. PGU and Meta-Unlearning turn out to be complementary: which performs better depends on how the concept is encoded, and retain concept selection should follow visual feature similarity rather than semantic grouping.
Key Contributions
- First adaptation of Projected Gradient Unlearning (PGU) from classification to text-to-image diffusion models as a post-hoc hardening step
- Efficient modular defense (6 minutes) that prevents concept revival during fine-tuning, compatible with existing unlearning methods (ESD, UCE, Receler)
- Empirical validation showing visual feature similarity (not semantic grouping) is key for retain concept selection, with complete revival elimination for style concepts
🛡️ Threat Analysis
The paper addresses machine unlearning as a defense against adversarial extraction of removed concepts. By preventing concept revival during fine-tuning, PGU defends against an adversary who could exploit fine-tuning to restore erased private/copyrighted content. The threat model involves an adversary attempting to recover 'forgotten' training data through downstream adaptation.