Can We Trust LLM Detectors?

The rapid adoption of LLMs has increased the need for reliable AI text detection, yet existing detectors often fail outside controlled benchmarks. We systematically evaluate 2 dominant paradigms (training-free and supervised) and show that both are brittle under distribution shift, unseen generators, and simple stylistic perturbations. To address these limitations, we propose a supervised contrastive learning (SCL) framework that learns discriminative style embeddings. Experiments show that while supervised detectors excel in-domain, they degrade sharply out-of-domain, and training-free methods remain highly sensitive to proxy choice. Overall, our results expose fundamental challenges in building domain-agnostic detectors. Our code is available at: https://github.com/HARSHITJAIS14/DetectAI

Key Contributions

• Systematic evaluation showing both training-free and supervised AI text detectors fail severely under distribution shift and unseen generators
• Supervised contrastive learning (SCL) framework using DeBERTa-v3 with InfoNCE loss that learns discriminative style embeddings and enables few-shot adaptation with as few as 25 examples
• Comprehensive adversarial and OOD robustness analysis demonstrating that no current paradigm achieves domain-agnostic detection

🛡️ Threat Analysis

Details

Similar Papers