ML Security PapersPromptCOS: Towards Content-only System Prompt Copyright Auditing for LLMs
Yuchen Yang 1, Yiming Li 2, Hongwei Yao 1, Enhao Huang 1, Shuo Shao 1, Yuyi Wang 3,4, Zhibo Wang 1, Dacheng Tao 2, Zhan Qin 1
Published on arXiv
2509.03117
Model Theft
OWASP ML Top 10 — ML05
Sensitive Information Disclosure
OWASP LLM Top 10 — LLM06
Key Finding
Achieves 99.3% average watermark similarity, 60.8% higher distinctiveness than the best baseline, accuracy degradation ≤ 0.6%, and up to 98.1% computational cost saving compared to prior logit-dependent methods.
PromptCOS
Novel technique introduced
System prompts are critical for shaping the behavior and output quality of large language model (LLM)-based applications, driving substantial investment in optimizing high-quality prompts beyond traditional handcrafted designs. However, as system prompts become valuable intellectual property, they are increasingly vulnerable to prompt theft and unauthorized use, highlighting the urgent need for effective copyright auditing, especially watermarking. Existing methods rely on verifying subtle logit distribution shifts triggered by a query. We observe that this logit-dependent verification framework is impractical in real-world content-only settings, primarily because (1) random sampling makes content-level generation unstable for verification, and (2) stronger instructions needed for content-level signals compromise prompt fidelity. To overcome these challenges, we propose PromptCOS, the first content-only system prompt copyright auditing method based on content-level output similarity. PromptCOS achieves watermark stability by designing a cyclic output signal as the conditional instruction's target. It preserves prompt fidelity by injecting a small set of auxiliary tokens to encode the watermark, leaving the main prompt untouched. Furthermore, to ensure robustness against malicious removal, we optimize cover tokens, i.e., critical tokens in the original prompt, to ensure that removing auxiliary tokens causes severe performance degradation. Experimental results show that PromptCOS achieves high effectiveness (99.3% average watermark similarity), strong distinctiveness (60.8% higher than the best baseline), high fidelity (accuracy degradation no greater than 0.6%), robustness (resilience against four potential attack categories), and high computational efficiency (up to 98.1% cost saving). Our code is available at GitHub (https://github.com/LianPing-cyber/PromptCOS).
Key Contributions
- First content-only system prompt watermarking method (PromptCOS) that operates without access to logits, using content-level output similarity for verification
- Cyclic output signal design for stable watermark verification under random sampling, paired with auxiliary token injection that leaves the main prompt intact to preserve fidelity
- Cover token optimization that causes severe performance degradation if the adversary removes the auxiliary watermark tokens, ensuring robustness against removal attacks
🛡️ Threat Analysis
System prompts are treated as valuable intellectual property, and PromptCOS embeds a watermark in the prompt (via auxiliary tokens) to prove ownership and detect unauthorized use — directly analogous to model watermarking for IP protection. The cover-token optimization ensures the watermark survives removal attempts, mirroring model watermark robustness requirements.