Privacy Enhanced PEFT: Tensor Train Decomposition Improves Privacy Utility Tradeoffs under DP-SGD

Fine-tuning large language models on sensitive data poses significant privacy risks, as membership inference attacks can reveal whether individual records were used during training. While Differential Privacy (DP) provides formal protection, applying DP to conventional Parameter-Efficient Fine-Tuning (PEFT) methods such as Low-Rank Adaptation (LoRA) often incurs substantial utility loss. In this work, we show that a more structurally constrained PEFT architecture, Tensor Train Low-Rank Adaptation (TTLoRA), can improve the privacy-utility tradeoff by shrinking the effective parameter space while preserving expressivity. To this end, we develop TTLoRA-DP, a differentially private training framework for TTLoRA. Specifically, we extend the ghost clipping algorithm to Tensor Train cores via cached contraction states, enabling efficient Differentially Private Stochastic Gradient Descent (DP-SGD) with exact per-example gradient norm computation without materializing full per-example gradients. Experiments on GPT-2 fine-tuning over the Enron and Penn Treebank datasets show that TTLoRA-DP consistently strengthens privacy protection relative to LoRA-DP while maintaining comparable or better downstream utility. Moreover, TTLoRA exhibits lower membership leakage even without DP training, using substantially smaller adapters and requiring on average 7.6X fewer parameters than LoRA. Overall, our results demonstrate that TTLoRA offers a practical path to improving the privacy-utility tradeoff in parameter-efficient language model adaptation.

Key Contributions

TTLoRA-DP: extends ghost clipping to tensor train cores via cached contraction states for efficient DP-SGD without materializing full per-example gradients
Demonstrates that tensor train decomposition improves the privacy-utility tradeoff over LoRA-DP with MIA AUC near random guessing across all epsilon budgets
Shows TTLoRA reduces membership leakage even without DP training using 7.6x fewer parameters than LoRA on average

🛡️ Threat Analysis

Membership Inference Attack

The paper's primary security contribution is defending against membership inference attacks during LLM fine-tuning; evaluation explicitly uses MIA AUC (PreCurious protocol) across epsilon budgets to show TTLoRA-DP keeps MIA near random guessing, directly targeting the binary 'was this record in training?' threat.