Erase or Hide? Suppressing Spurious Unlearning Neurons for Robust Unlearning

Large language models trained on web-scale data can memorize private or sensitive knowledge, raising significant privacy risks. Although some unlearning methods mitigate these risks, they remain vulnerable to "relearning" during subsequent training, allowing a substantial portion of forgotten knowledge to resurface. In this paper, we show that widely used unlearning methods cause shallow alignment: instead of faithfully erasing target knowledge, they generate spurious unlearning neurons that amplify negative influence to hide it. To overcome this limitation, we introduce Ssiuu, a new class of unlearning methods that employs attribution-guided regularization to prevent spurious negative influence and faithfully remove target knowledge. Experimental results confirm that our method reliably erases target knowledge and outperforms strong baselines across two practical retraining scenarios: (1) adversarial injection of private data, and (2) benign attack using an instruction-following benchmark. Our findings highlight the necessity of robust and faithful unlearning methods for safe deployment of language models.

Key Contributions

Identifies 'shallow unlearning alignment' — existing unlearning methods create spurious neurons that hide rather than erase target knowledge, making them vulnerable to relearning
Proposes Ssiuu, an attribution-guided regularization method that suppresses spurious negative influence to achieve faithful knowledge erasure
Demonstrates robustness against two adversarial relearning scenarios: adversarial injection of private data and benign instruction-following fine-tuning