ML Security PapersHow Do Semantically Equivalent Code Transformations Impact Membership Inference on LLMs for Code?
Hua Yang 1, Alejandro Velasco 2, Thanh Le-Cong 3, Md Nazmul Haque 1, Bowen Xu 1, Denys Poshyvanyk 2
Published on arXiv
2512.15468
Membership Inference Attack
OWASP ML Top 10 — ML04
Key Finding
RenameVariable transformation reduces MI detection success by 10.19% while incurring only 1.5% model accuracy drop, exposing a critical evasion loophole in MI-based license compliance enforcement for code LLMs.
The success of large language models for code relies on vast amounts of code data, including public open-source repositories, such as GitHub, and private, confidential code from companies. This raises concerns about intellectual property compliance and the potential unauthorized use of license-restricted code. While membership inference (MI) techniques have been proposed to detect such unauthorized usage, their effectiveness can be undermined by semantically equivalent code transformation techniques, which modify code syntax while preserving semantic. In this work, we systematically investigate whether semantically equivalent code transformation rules might be leveraged to evade MI detection. The results reveal that model accuracy drops by only 1.5% in the worst case for each rule, demonstrating that transformed datasets can effectively serve as substitutes for fine-tuning. Additionally, we find that one of the rules (RenameVariable) reduces MI success by 10.19%, highlighting its potential to obscure the presence of restricted code. To validate these findings, we conduct a causal analysis confirming that variable renaming has the strongest causal effect in disrupting MI detection. Notably, we find that combining multiple transformations does not further reduce MI effectiveness. Our results expose a critical loophole in license compliance enforcement for training large language models for code, showing that MI detection can be substantially weakened by transformation-based obfuscation techniques.
Key Contributions
- Systematic empirical investigation of how semantically equivalent code transformations affect membership inference effectiveness on LLMs for code
- Identification of RenameVariable as the single most effective transformation for reducing MI success (10.19% reduction) while preserving fine-tuning utility (only 1.5% accuracy drop)
- Causal analysis confirming variable renaming has the strongest causal effect on disrupting MI detection, and that combining multiple transformations yields no additional benefit
🛡️ Threat Analysis
The paper's primary focus is membership inference attacks on LLMs — specifically whether code transformations can be used to evade MI detection, thereby undermining license compliance enforcement that relies on MI techniques.