ML Security PapersUnderstanding Privacy Risks in Code Models Through Training Dynamics: A Causal Approach
Hua Yang 1, Alejandro Velasco 2, Sen Fang 1, Bowen Xu 1, Denys Poshyvanyk 2
Published on arXiv
2512.07814
Model Inversion Attack
OWASP ML Top 10 — ML03
Sensitive Information Disclosure
OWASP LLM Top 10 — LLM06
Key Finding
Causal analysis confirms leakage risk is type-dependent: IP addresses (easy-to-learn) leak significantly more than keys and passwords (hard-to-learn) across LLM4Code models of varying scales.
Training Dynamics Causal Model for PII Leakage
Novel technique introduced
Large language models for code (LLM4Code) have greatly improved developer productivity but also raise privacy concerns due to their reliance on open-source repositories containing abundant personally identifiable information (PII). Prior work shows that commercial models can reproduce sensitive PII, yet existing studies largely treat PII as a single category and overlook the heterogeneous risks among different types. We investigate whether distinct PII types vary in their likelihood of being learned and leaked by LLM4Code, and whether this relationship is causal. Our methodology includes building a dataset with diverse PII types, fine-tuning representative models of different scales, computing training dynamics on real PII data, and formulating a structural causal model to estimate the causal effect of learnability on leakage. Results show that leakage risks differ substantially across PII types and correlate with their training dynamics: easy-to-learn instances such as IP addresses exhibit higher leakage, while harder types such as keys and passwords leak less frequently. Ambiguous types show mixed behaviors. This work provides the first causal evidence that leakage risks are type-dependent and offers guidance for developing type-aware and learnability-aware defenses for LLM4Code.
Key Contributions
- First causal evidence (via structural causal model) that PII leakage risk in LLM4Code is type-dependent and causally linked to learnability as measured by training dynamics
- A curated PII dataset covering diverse PII types extracted from open-source repositories, enabling systematic leakage evaluation across PII categories
- Empirical finding that easy-to-learn PII (e.g., IP addresses) exhibits substantially higher leakage risk than hard-to-learn types (e.g., keys, passwords), with guidance for type-aware defenses
🛡️ Threat Analysis
The paper measures and causally explains how LLM4Code models memorize and leak private training data (PII); 'LLM memorization extraction' is explicitly listed under ML03, and the adversary threat model is someone querying models to reproduce sensitive training data verbatim.