TY - GEN
T1 - Binary Black-Box Attacks against Static Malware Detectors with Reinforcement Learning in Discrete Action Spaces
AU - Ebrahimi, Mohammadreza
AU - Pacheco, Jason
AU - Li, Weifeng
AU - Hu, James Lee
AU - Chen, Hsinchun
N1 - Funding Information:
This material is based upon work supported by the National Science Foundation (NSF) under the grants SaTC-1936370 and SFS-1921485. We would like to thank VirusTotal for providing the malware dataset and granting access to the APIs for malware functionality assessment.
Publisher Copyright:
© 2021 IEEE.
PY - 2021/5
Y1 - 2021/5
N2 - Recent machine learning- and deep learning-based static malware detectors have shown breakthrough performance in identifying unseen malware variants. As a result, they are increasingly being adopted to lower the cost of dynamic malware analysis and manual signature identification. Despite their success, studies have shown that they can be vulnerable to adversarial malware attacks, in which an adversary modifies a known malware executable subtly to fool the malware detector into recognizing it as a benign file. Recent studies have shown that automatically crafting these adversarial malware variants at scale is beneficial to improve the robustness of malware detectors. For conciseness, we refer to this process as Adversarial Malware example Generation (AMG). Most AMG methods rely on prior knowledge about the architecture or parameters of the detector, which is not often available in practice. Moreover, the majority of these methods are restricted to additive modifications that append contents to the malware executable without modifying its original content. In this study, we offer a novel Reinforcement Learning (RL) method, AMG-VAC, which extends Variational Actor-Critic (VAC) to non-continuous action spaces where modifications are inherently discrete. We evaluate the evasion performance of the proposed AMG-VAC on two reputable machine learning-based malware detectors. While the proposed method outperforms extant non-RL and RL-based AMG methods by statistically significant margins, we show that the obtained evasive action sequences are useful in shedding light on malware detectors' vulnerabilities.
AB - Recent machine learning- and deep learning-based static malware detectors have shown breakthrough performance in identifying unseen malware variants. As a result, they are increasingly being adopted to lower the cost of dynamic malware analysis and manual signature identification. Despite their success, studies have shown that they can be vulnerable to adversarial malware attacks, in which an adversary modifies a known malware executable subtly to fool the malware detector into recognizing it as a benign file. Recent studies have shown that automatically crafting these adversarial malware variants at scale is beneficial to improve the robustness of malware detectors. For conciseness, we refer to this process as Adversarial Malware example Generation (AMG). Most AMG methods rely on prior knowledge about the architecture or parameters of the detector, which is not often available in practice. Moreover, the majority of these methods are restricted to additive modifications that append contents to the malware executable without modifying its original content. In this study, we offer a novel Reinforcement Learning (RL) method, AMG-VAC, which extends Variational Actor-Critic (VAC) to non-continuous action spaces where modifications are inherently discrete. We evaluate the evasion performance of the proposed AMG-VAC on two reputable machine learning-based malware detectors. While the proposed method outperforms extant non-RL and RL-based AMG methods by statistically significant margins, we show that the obtained evasive action sequences are useful in shedding light on malware detectors' vulnerabilities.
KW - adversarial malware generation
KW - approximate sampling
KW - binary black-box attack
KW - reinforcement learning
KW - static malware detection
KW - variational actor-critic
UR - http://www.scopus.com/inward/record.url?scp=85112856144&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85112856144&partnerID=8YFLogxK
U2 - 10.1109/SPW53761.2021.00021
DO - 10.1109/SPW53761.2021.00021
M3 - Conference contribution
AN - SCOPUS:85112856144
T3 - Proceedings - 2021 IEEE Symposium on Security and Privacy Workshops, SPW 2021
SP - 85
EP - 91
BT - Proceedings - 2021 IEEE Symposium on Security and Privacy Workshops, SPW 2021
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 2021 IEEE Symposium on Security and Privacy Workshops, SPW 2021
Y2 - 27 May 2021
ER -