ML Security PapersA Practical Adversarial Attack against Sequence-based Deep Learning Malware Classifiers
Kai Tan , Dongyang Zhan , Lin Ye , Hongli Zhang , Binxing Fang
Published on arXiv
2509.11836
Input Manipulation Attack
OWASP ML Top 10 — ML01
Key Finding
Achieves high attack success rate against sequence-based anomaly detection models while preserving malware functionality and satisfying real-world behavioral constraints.
DQN-based adversarial malware perturbation with heuristic backtracking
Novel technique introduced
Sequence-based deep learning models (e.g., RNNs), can detect malware by analyzing its behavioral sequences. Meanwhile, these models are susceptible to adversarial attacks. Attackers can create adversarial samples that alter the sequence characteristics of behavior sequences to deceive malware classifiers. The existing methods for generating adversarial samples typically involve deleting or replacing crucial behaviors in the original data sequences, or inserting benign behaviors that may violate the behavior constraints. However, these methods that directly manipulate sequences make adversarial samples difficult to implement or apply in practice. In this paper, we propose an adversarial attack approach based on Deep Q-Network and a heuristic backtracking search strategy, which can generate perturbation sequences that satisfy practical conditions for successful attacks. Subsequently, we utilize a novel transformation approach that maps modifications back to the source code, thereby avoiding the need to directly modify the behavior log sequences. We conduct an evaluation of our approach, and the results confirm its effectiveness in generating adversarial samples from real-world malware behavior sequences, which have a high success rate in evading anomaly detection models. Furthermore, our approach is practical and can generate adversarial samples while maintaining the functionality of the modified software.
Key Contributions
- DQN-based adversarial perturbation generator that selects optimal benign behavior insertions to minimally perturb malware behavioral sequences and evade RNN-based classifiers
- Heuristic backtracking search strategy ensuring generated perturbation sequences satisfy practical behavioral constraints (e.g., valid API call ordering)
- Novel transformation approach that maps sequence-level modifications back to malware source code, enabling practical real-world attacks without directly tampering with behavior logs
🛡️ Threat Analysis
Proposes a method to craft adversarial inputs (modified malware behavioral sequences) that cause misclassification by sequence-based deep learning malware detectors at inference time — core adversarial example attack, using RL (DQN) rather than gradients to optimize perturbations.