ML Security PapersWho Stole Your Data? A Method for Detecting Unauthorized RAG Theft
Peiyang Liu 1, Ziqiang Cui 2, Di Liang 3, Wei Ye 1
Published on arXiv
2510.07728
Output Integrity Attack
OWASP ML Top 10 — ML09
Key Finding
The dual-layered watermarking approach successfully detects unauthorized RAG usage across varying query volumes, defense prompts, and retrieval parameters, demonstrating resilience against adversarial evasion techniques while maintaining high output text quality.
Dual-Layered RAG Watermarking with Interrogator-Detective Framework
Novel technique introduced
Retrieval-augmented generation (RAG) enhances Large Language Models (LLMs) by mitigating hallucinations and outdated information issues, yet simultaneously facilitates unauthorized data appropriation at scale. This paper addresses this challenge through two key contributions. First, we introduce RPD, a novel dataset specifically designed for RAG plagiarism detection that encompasses diverse professional domains and writing styles, overcoming limitations in existing resources. Second, we develop a dual-layered watermarking system that embeds protection at both semantic and lexical levels, complemented by an interrogator-detective framework that employs statistical hypothesis testing on accumulated evidence. Extensive experimentation demonstrates our approach's effectiveness across varying query volumes, defense prompts, and retrieval parameters, while maintaining resilience against adversarial evasion techniques. This work establishes a foundational framework for intellectual property protection in retrieval-augmented AI systems.
Key Contributions
- RPD: a novel dataset for RAG plagiarism detection covering diverse professional domains and writing styles to realistically simulate RAG-generated content
- Dual-layered watermarking system embedding protection at both semantic and lexical levels to survive LLM reformulation of retrieved content
- Interrogator-Detective framework that strategically crafts queries to accumulate watermark evidence and applies statistical hypothesis testing for reliable unauthorized-use detection
🛡️ Threat Analysis
The paper proposes embedding watermarks in documents (content) so that if an unauthorized RAG system uses them, the marks propagate into generated outputs and can be detected via statistical hypothesis testing — this is content watermarking for provenance and IP protection, analogous to watermarking training data to detect misappropriation.