ML Security PapersBudgetLeak: Membership Inference Attacks on RAG Systems via the Generation Budget Side Channel
Hao Li 1, Jiajun He 1, Guangshuo Wang 1, Dengguo Feng 1, Zheng Li 2, Min Zhang 1
Published on arXiv
2511.12043
Membership Inference Attack
OWASP ML Top 10 — ML04
Sensitive Information Disclosure
OWASP LLM Top 10 — LLM06
Key Finding
BudgetLeak consistently outperforms existing MIA baselines across four datasets, three LLM generators, and two retrievers by exploiting generation budget variance as a membership signal.
BudgetLeak
Novel technique introduced
Retrieval-Augmented Generation (RAG) enhances large language models by integrating external knowledge, but reliance on proprietary or sensitive corpora poses various data risks, including privacy leakage and unauthorized data usage. Membership inference attacks (MIAs) are a common technique to assess such risks, yet existing approaches underperform in RAG due to black-box constraints and the absence of strong membership signals. In this paper, we identify a previously unexplored side channel in RAG systems: the generation budget, which controls the maximum number of tokens allowed in a generated response. Varying this budget reveals observable behavioral patterns between member and non-member queries, as members gain quality more rapidly with larger budgets. Building on this insight, we propose BudgetLeak, a novel membership inference attack that probes responses under different budgets and analyzes metric evolution via sequence modeling or clustering. Extensive experiments across four datasets, three LLM generators, and two retrievers demonstrate that BudgetLeak consistently outperforms existing baselines, while maintaining high efficiency and practical viability. Our findings reveal a previously overlooked data risk in RAG systems and highlight the need for new defenses.
Key Contributions
- Identifies a previously unexplored side channel in RAG systems — the generation budget — that reveals differential behavioral patterns between member and non-member queries.
- Proposes BudgetLeak, a MIA that probes RAG responses across multiple token budgets and analyzes metric evolution via sequence modeling or clustering to infer membership.
- Demonstrates consistent outperformance over existing MIA baselines across four datasets, three LLM generators, and two retrievers under black-box constraints.
🛡️ Threat Analysis
BudgetLeak is a membership inference attack — its primary contribution is determining whether a specific data sample was ingested into a RAG system's knowledge base, the canonical ML04 task.