DistillGuard: Evaluating Defenses Against LLM Knowledge Distillation

Knowledge distillation from proprietary LLM APIs poses a growing threat to model providers, yet defenses against this attack remain fragmented and unevaluated. We present DistillGuard, a framework for systematically evaluating output-level defenses against LLM knowledge distillation. We introduce a taxonomy of three defense categories -- output perturbation, data poisoning, and information throttling -- and evaluate nine defense configurations using a standardized pipeline with Qwen3-14B as teacher and Qwen2.5-7B-Instruct as student across three benchmarks (MATH-500, HumanEval+, MT-Bench). Our results reveal that, in a same-family distillation setting against a naive attacker, most output-level defenses are surprisingly ineffective: paraphrasing-based perturbation barely degrades distilled student quality, and data poisoning primarily impairs conversational fluency while leaving task-specific capabilities intact. Only chain-of-thought removal substantially impairs mathematical reasoning (31.4\% vs.\ 67.8\% baseline), though code generation remains unaffected. These findings demonstrate that the effectiveness of distillation defenses is highly task-dependent and that current output-level approaches are insufficient to broadly prevent knowledge theft.

Key Contributions

• Taxonomy of output-level distillation defenses into three categories: output perturbation, data poisoning, and information throttling
• Standardized evaluation framework (DistillGuard) with metrics for distillation effectiveness and distillation cost across nine defense configurations
• Empirical finding that most output-level defenses are insufficient: only chain-of-thought removal substantially impairs distillation quality, and only for reasoning-heavy tasks

🛡️ Threat Analysis

Details

Similar Papers