Exploring Membership Inference Vulnerabilities in Clinical Large Language Models

As large language models (LLMs) become progressively more embedded in clinical decision-support, documentation, and patient-information systems, ensuring their privacy and trustworthiness has emerged as an imperative challenge for the healthcare sector. Fine-tuning LLMs on sensitive electronic health record (EHR) data improves domain alignment but also raises the risk of exposing patient information through model behaviors. In this work-in-progress, we present an exploratory empirical study on membership inference vulnerabilities in clinical LLMs, focusing on whether adversaries can infer if specific patient records were used during model training. Using a state-of-the-art clinical question-answering model, Llemr, we evaluate both canonical loss-based attacks and a domain-motivated paraphrasing-based perturbation strategy that more realistically reflects clinical adversarial conditions. Our preliminary findings reveal limited but measurable membership leakage, suggesting that current clinical LLMs provide partial resistance yet remain susceptible to subtle privacy risks that could undermine trust in clinical AI adoption. These results motivate continued development of context-aware, domain-specific privacy evaluations and defenses such as differential privacy fine-tuning and paraphrase-aware training, to strengthen the security and trustworthiness of healthcare AI systems.

Key Contributions

• Empirical evaluation of canonical loss-based membership inference attacks on Llemr, a clinical QA LLM fine-tuned on EHR data
• Novel domain-motivated paraphrasing-based perturbation strategy for MIA that more realistically models clinical adversarial conditions
• Preliminary evidence of limited but measurable membership leakage in clinical LLMs, motivating domain-specific defenses such as differential privacy fine-tuning and paraphrase-aware training

🛡️ Threat Analysis

Details

Similar Papers