Performance Decay in Deepfake Detection: The Limitations of Training on Outdated Data

The continually advancing quality of deepfake technology exacerbates the threats of disinformation, fraud, and harassment by making maliciously-generated synthetic content increasingly difficult to distinguish from reality. We introduce a simple yet effective two-stage detection method that achieves an AUROC of over 99.8% on contemporary deepfakes. However, this high performance is short-lived. We show that models trained on this data suffer a recall drop of over 30% when evaluated on deepfakes created with generation techniques from just six months later, demonstrating significant decay as threats evolve. Our analysis reveals two key insights for robust detection. Firstly, continued performance requires the ongoing curation of large, diverse datasets. Second, predictive power comes primarily from static, frame-level artifacts, not temporal inconsistencies. The future of effective deepfake detection therefore depends on rapid data collection and the development of advanced frame-level feature detectors.

Key Contributions

• Two-stage deepfake detection method achieving >99.8% AUROC on contemporary deepfakes
• Empirical demonstration that detection models suffer >30% recall drop within six months when evaluated on deepfakes created with newer generation techniques
• Finding that frame-level static artifacts carry more predictive power than temporal inconsistencies for robust deepfake detection

🛡️ Threat Analysis

Details

Similar Papers