ML Security PapersSynBench: A Benchmark for Differentially Private Text Generation
Yidan Sun 1, Viktor Schlegel 2, Srinivasan Nandakumar 1,2, Iqra Zahid 2, Yuping Wu 2, Yulong Wu 1, Hao Li 2, Jie Zhang 3, Warren Del-Pinto 4, Goran Nenadic 1, Siew Kei Lam 2, Anil Anthony Bharath 1
Published on arXiv
2509.14594
Membership Inference Attack
OWASP ML Top 10 — ML04
Key Finding
Membership inference attacks on DP-generated synthetic text reveal that pre-training contamination can invalidate claimed DP privacy guarantees, with domain complexity being the primary predictor of generation quality degradation.
SynBench
Novel technique introduced
Data-driven decision support in high-stakes domains like healthcare and finance faces significant barriers to data sharing due to regulatory, institutional, and privacy concerns. While recent generative AI models, such as large language models, have shown impressive performance in open-domain tasks, their adoption in sensitive environments remains limited by unpredictable behaviors and insufficient privacy-preserving datasets for benchmarking. Existing anonymization methods are often inadequate, especially for unstructured text, as redaction and masking can still allow re-identification. Differential Privacy (DP) offers a principled alternative, enabling the generation of synthetic data with formal privacy assurances. In this work, we address these challenges through three key contributions. First, we introduce a comprehensive evaluation framework with standardized utility and fidelity metrics, encompassing nine curated datasets that capture domain-specific complexities such as technical jargon, long-context dependencies, and specialized document structures. Second, we conduct a large-scale empirical study benchmarking state-of-the-art DP text generation methods and LLMs of varying sizes and different fine-tuning strategies, revealing that high-quality domain-specific synthetic data generation under DP constraints remains an unsolved challenge, with performance degrading as domain complexity increases. Third, we develop a membership inference attack (MIA) methodology tailored for synthetic text, providing first empirical evidence that the use of public datasets - potentially present in pre-training corpora - can invalidate claimed privacy guarantees. Our findings underscore the urgent need for rigorous privacy auditing and highlight persistent gaps between open-domain and specialist evaluations, informing responsible deployment of generative AI in privacy-sensitive, high-stakes settings.
Key Contributions
- SynBench: a standardized evaluation framework with utility/fidelity metrics across nine domain-specific datasets for benchmarking DP text generation methods
- Large-scale empirical study revealing that high-quality domain-specific DP synthetic text generation remains unsolved, with performance degrading as domain complexity increases
- MIA methodology for synthetic text demonstrating that public datasets present in LLM pre-training corpora can invalidate formally claimed differential privacy guarantees
🛡️ Threat Analysis
The paper's third contribution is a novel membership inference attack methodology specifically designed for synthetic text, providing empirical evidence that pre-training data contamination allows an adversary to invalidate claimed differential privacy guarantees — a direct membership inference threat against LLM-based DP generative models.