ML Security PapersDCMI: A Differential Calibration Membership Inference Attack Against Retrieval-Augmented Generation
Xinyu Gao , Xiangtao Meng , Yingkai Dong , Zheng Li , Shanqing Guo
Published on arXiv
2509.06026
Membership Inference Attack
OWASP ML Top 10 — ML04
Sensitive Information Disclosure
OWASP LLM Top 10 — LLM06
Key Finding
DCMI achieves 97.42% AUC and 94.35% accuracy against Flan-T5-based RAG, exceeding the MBA baseline by over 40%, with 10–20% gains on real-world RAG platforms
DCMI
Novel technique introduced
While Retrieval-Augmented Generation (RAG) effectively reduces hallucinations by integrating external knowledge bases, it introduces vulnerabilities to membership inference attacks (MIAs), particularly in systems handling sensitive data. Existing MIAs targeting RAG's external databases often rely on model responses but ignore the interference of non-member-retrieved documents on RAG outputs, limiting their effectiveness. To address this, we propose DCMI, a differential calibration MIA that mitigates the negative impact of non-member-retrieved documents. Specifically, DCMI leverages the sensitivity gap between member and non-member retrieved documents under query perturbation. It generates perturbed queries for calibration to isolate the contribution of member-retrieved documents while minimizing the interference from non-member-retrieved documents. Experiments under progressively relaxed assumptions show that DCMI consistently outperforms baselines--for example, achieving 97.42% AUC and 94.35% Accuracy against the RAG system with Flan-T5, exceeding the MBA baseline by over 40%. Furthermore, on real-world RAG platforms such as Dify and MaxKB, DCMI maintains a 10%-20% advantage over the baseline. These results highlight significant privacy risks in RAG systems and emphasize the need for stronger protection mechanisms. We appeal to the community's consideration of deeper investigations, like ours, against the data leakage risks in rapidly evolving RAG systems. Our code is available at https://github.com/Xinyu140203/RAG_MIA.
Key Contributions
- Identifies that non-member retrieved documents interfere with and degrade existing MIAs targeting RAG knowledge bases
- Proposes DCMI which exploits the sensitivity gap between member and non-member documents under query perturbation to isolate member-document contribution signals
- Demonstrates 97.42% AUC on Flan-T5 RAG systems (40%+ over MBA baseline) and 10–20% advantage on real-world platforms Dify and MaxKB
🛡️ Threat Analysis
Core contribution is a membership inference attack that determines whether a specific document is in a RAG system's retrieval database — the canonical ML04 threat, applied to RAG external knowledge stores rather than model training sets.