ML Security PapersRethinking Cross-Domain Evaluation for Face Forgery Detection with Semantic Fine-grained Alignment and Mixture-of-Experts
Yuhan Luo , Tao Chen , Decheng Liu
Published on arXiv
2604.21478
Output Integrity Attack
OWASP ML Top 10 — ML09
Key Finding
Achieves superior cross-domain generalization performance compared to SOTA methods under both standard AUC and the proposed Cross-AUC metric
SFAM
Novel technique introduced
Nowadays, visual data forgery detection plays an increasingly important role in social and economic security with the rapid development of generative models. Existing face forgery detectors still can't achieve satisfactory performance because of poor generalization ability across datasets. The key factor that led to this phenomenon is the lack of suitable metrics: the commonly used cross-dataset AUC metric fails to reveal an important issue where detection scores may shift significantly across data domains. To explicitly evaluate cross-domain score comparability, we propose \textbf{Cross-AUC}, an evaluation metric that can compute AUC across dataset pairs by contrasting real samples from one dataset with fake samples from another (and vice versa). It is interesting to find that evaluating representative detectors under the Cross-AUC metric reveals substantial performance drops, exposing an overlooked robustness problem. Besides, we also propose the novel framework \textbf{S}emantic \textbf{F}ine-grained \textbf{A}lignment and \textbf{M}ixture-of-Experts (\textbf{SFAM}), consisting of a patch-level image-text alignment module that enhances CLIP's sensitivity to manipulation artifacts, and the facial region mixture-of-experts module, which routes features from different facial regions to specialized experts for region-aware forgery analysis. Extensive qualitative and quantitative experiments on the public datasets prove that the proposed method achieves superior performance compared with the state-of-the-art methods with various suitable metrics.
Key Contributions
- Cross-AUC metric that evaluates cross-domain score comparability by computing AUC across dataset pairs, revealing performance drops in existing detectors
- SFAM framework combining patch-level image-text alignment (PaITA) to enhance CLIP's sensitivity to manipulation artifacts
- Facial region mixture-of-experts (FaRMoE) module routing features from different facial regions to specialized experts for region-aware forgery analysis
🛡️ Threat Analysis
Detects AI-generated/manipulated facial content (deepfakes) to verify authenticity — this is output integrity and content provenance, the core function of deepfake detection.