DeepProv: Behavioral Characterization and Repair of Neural Networks via Inference Provenance Graph Analysis

Deep neural networks (DNNs) are increasingly being deployed in high-stakes applications, from self-driving cars to biometric authentication. However, their unpredictable and unreliable behaviors in real-world settings require new approaches to characterize and ensure their reliability. This paper introduces DeepProv, a novel and customizable system designed to capture and characterize the runtime behavior of DNNs during inference by using their underlying graph structure. Inspired by system audit provenance graphs, DeepProv models the computational information flow of a DNN's inference process through Inference Provenance Graphs (IPGs). These graphs provide a detailed structural representation of the behavior of DNN, allowing both empirical and structural analysis. DeepProv uses these insights to systematically repair DNNs for specific objectives, such as improving robustness, privacy, or fairness. We instantiate DeepProv with adversarial robustness as the goal of model repair and conduct extensive case studies to evaluate its effectiveness. Our results demonstrate its effectiveness and scalability across diverse classification tasks, attack scenarios, and model complexities. DeepProv automatically identifies repair actions at the node and edge-level within IPGs, significantly enhancing the robustness of the model. In particular, applying DeepProv repair strategies to just a single layer of a DNN yields an average 55% improvement in adversarial accuracy. Moreover, DeepProv complements existing defenses, achieving substantial gains in adversarial robustness. Beyond robustness, we demonstrate the broader potential of DeepProv as an adaptable system to characterize DNN behavior in other critical areas, such as privacy auditing and fairness analysis.

Key Contributions

Inference Provenance Graphs (IPGs) that model computational information flow through a DNN during inference, enabling structural behavioral analysis
DeepProv system that automatically identifies and repairs vulnerable nodes/edges within IPGs to improve adversarial robustness, privacy, or fairness
Model repair applied to a single DNN layer achieves an average 55% improvement in adversarial accuracy and complements existing defenses

🛡️ Threat Analysis

Input Manipulation Attack

The paper's primary instantiation and evaluation is defense against adversarial examples — it repairs DNN layers using IPG analysis to improve adversarial accuracy by 55%, directly defending against inference-time input manipulation attacks.