Signature vs. Substance: Evaluating the Balance of Adversarial Resistance and Linguistic Quality in Watermarking Large Language Models

To mitigate the potential harms of Large Language Models (LLMs)generated text, researchers have proposed watermarking, a process of embedding detectable signals within text. With watermarking, we can always accurately detect LLM-generated texts. However, recent findings suggest that these techniques often negatively affect the quality of the generated texts, and adversarial attacks can strip the watermarking signals, causing the texts to possibly evade detection. These findings have created resistance in the wide adoption of watermarking by LLM creators. Finally, to encourage adoption, we evaluate the robustness of several watermarking techniques to adversarial attacks by comparing paraphrasing and back translation (i.e., English $\to$ another language $\to$ English) attacks; and their ability to preserve quality and writing style of the unwatermarked texts by using linguistic metrics to capture quality and writing style of texts. Our results suggest that these watermarking techniques preserve semantics, deviate from the writing style of the unwatermarked texts, and are susceptible to adversarial attacks, especially for the back translation attack.

Key Contributions

• Comparative robustness evaluation of four LLM text watermarking methods (KGW, SIR, EWD, Unbiased Watermarking) against paraphrasing and back-translation adversarial attacks
• Linguistic quality analysis measuring how watermarking deviates from unwatermarked text in semantics and writing style
• Finding that back-translation is more effective than paraphrasing at stripping watermarks, and watermarked texts preserve semantics but deviate in style

🛡️ Threat Analysis

Details

Similar Papers