ML Security PapersPrivUn: Unveiling Latent Ripple Effects and Shallow Forgetting in Privacy Unlearning
Xiaoyi Chen 1, Haoyuan Wang 2, Siyuan Tang 1, Sijia Liu 3, Liya Su 4, XiaoFeng Wang 1, Haixu Tang 5
Published on arXiv
2604.22076
Model Inversion Attack
OWASP ML Top 10 — ML03
Sensitive Information Disclosure
OWASP LLM Top 10 — LLM06
Key Finding
Systematic comparison of 19 unlearning methods reveals training pipeline manipulation outperforms data manipulation, with gradient similarity showing Pearson r=0.73 correlation to forgetting effectiveness
PrivUn
Novel technique introduced
Large language models (LLMs) often memorize private information during training, raising serious privacy concerns. While machine unlearning has emerged as a promising solution, its true effectiveness against privacy attacks remains unclear. To address this, we propose PrivUn, a new evaluation framework that systematically assesses unlearning robustness through three-tier attack scenarios: direct retrieval, in-context learning recovery, and fine-tuning restoration; combined with quantitative analysis using forgetting scores, association metrics, and forgetting depth assessment. Our study exposes significant weaknesses in current unlearning methods, revealing two key findings: 1) unlearning exhibits gradient-driven ripple effects: unlike traditional forgetting which follows semantic relations (e.g., knowledge graphs), privacy unlearning propagates across latent gradient-based associations; and 2) most methods suffer from shallow forgetting, failing to remove private information distributed across multiple deep model layers. To validate these insights, we explore two strategies: association-aware core-set selection that leverages gradient similarity, and multi-layer deep intervention through representational constraints. These strategies represent a paradigm shift from shallow forgetting to deep forgetting.
Key Contributions
- Three-tier evaluation framework (PrivUn) assessing unlearning robustness against direct retrieval, in-context learning recovery, and fine-tuning restoration attacks
- Quantitative metrics including forgetting scores, gradient-based association metrics, and layer-wise forgetting depth analysis via CKA
- Identification of gradient-driven ripple effects in privacy unlearning and shallow forgetting problem across model layers
🛡️ Threat Analysis
Paper evaluates unlearning methods against privacy attacks that extract memorized PII from LLMs. The three-tier attack scenarios (direct retrieval, ICL recovery, fine-tuning restoration) are all methods for reconstructing private training data. The adversary threat model is explicit: active attackers extracting 'forgotten' private information.