ML Security PapersDWBench: Holistic Evaluation of Watermark for Dataset Copyright Auditing
Xiao Ren 1, Xinyi Yu 2, Linkang Du 2, Min Chen 3, Yuanchao Shu 1, Zhou Su 2, Yunjun Gao 1, Zhikun Zhang 1
Published on arXiv
2602.13541
Output Integrity Attack
OWASP ML Top 10 — ML09
Key Finding
No single dataset watermarking method dominates all scenarios; existing techniques are unstable at low watermark rates and in multi-user settings, exhibiting elevated false positives or performance degradation.
DWBench
Novel technique introduced
The surging demand for large-scale datasets in deep learning has heightened the need for effective copyright protection, given the risks of unauthorized use to data owners. Although the dataset watermark technique holds promise for auditing and verifying usage, existing methods are hindered by inconsistent evaluations, which impede fair comparisons and assessments of real-world viability. To address this gap, we propose a two-layer taxonomy that categorizes methods by implementation (model-based vs. model-free injection; model-behavior vs. model-message verification), offering a structured framework for cross-task analysis. Then, we develop DWBench, a unified benchmark and open-source toolkit for systematically evaluating image dataset watermark techniques in classification and generation tasks. Using DWBench, we assess 25 representative methods under standardized conditions, perturbation-based robustness tests, multi-watermark coexistence, and multi-user interference. In addition to reporting the results of four commonly used metrics, we present the results of two new metrics: sample significance for fine-grained watermark distinguishability and verification success rate for dataset-level auditing, which enable accurate and reproducible benchmarking. Key findings reveal inherent trade-offs: no single method dominates all scenarios; classification and generation tasks require specialized approaches; and existing techniques exhibit instability at low watermark rates and in realistic multi-user settings, with elevated false positives or performance declines. We hope that DWBench can facilitate advances in watermark reliability and practicality, thus strengthening copyright safeguards in the face of widespread AI-driven data exploitation.
Key Contributions
- Two-layer taxonomy categorizing dataset watermark methods by injection (model-based vs. model-free) and verification (model-behavior vs. model-message) phases
- DWBench: open-source unified benchmark evaluating 25 representative methods across 6 datasets and 6 models under standardized, robustness, multi-watermark, and multi-user conditions
- Two new metrics — sample significance for fine-grained watermark distinguishability and verification success rate for dataset-level auditing — enabling more accurate reproducible comparison
🛡️ Threat Analysis
Dataset watermarks are embedded in training data images and verified through model outputs/behavior to detect unauthorized dataset usage — this is training data watermarking for content provenance and copyright auditing, not model weight protection (ML05). The benchmark evaluates 25 such techniques under robustness, multi-watermark coexistence, and multi-user interference conditions.