TY - GEN
T1 - An Adversarial Reinforcement Learning Framework for Robust Machine Learning-based Malware Detection
AU - Ebrahimi, Mohammadreza Reza
AU - Li, Weifeng
AU - Chai, Yidong
AU - Pacheco, Jason
AU - Chen, Hsinchun
N1 - Publisher Copyright:
© 2022 IEEE.
PY - 2022
Y1 - 2022
N2 - Empowered by the recent development in Ma-chine Learning (ML), signatureless ML-based malware detectors present promising performance in identifying unseen mal ware variants and zero days without requiring expensive dynamic malware analysis. However, it has been recently shown that ML-based malware detectors are vulnerable to adversarial malware attacks, in which an attacker modifies a known malware exe-cutable to trick the malware detector into recognizing the modi-fied variant as benign. Adversarial malware example generation has become an emerging area in adversarial ML that studies creating functionality-preserving adversarial malware variants. Advancements in this area have led to an eternal game between the adversary and defender. While the area has attracted much attention in the security community, a large body of these studies merely focuses on attack methods against ML-based malware detectors. There has been little work on understanding how these adversarial variants can be systematically used by the defender to strengthen the robustness of these detectors and stand ahead of the adversary. Latest efforts have led to emergence of adversarial learning. In this work, we propose a simple wargame approach to empirically conduct the adversarial minimax optimization underlying in the adversarial learning for improving the robustness of ML-based malware detectors. Our proposed approach employs adversarial malware variants generated from a reinforcement learning-based adversarial attack policy in a minimax game alternating between strengthening the attack policy and improving the detectors' robustness. We evaluated the effectiveness of our approach on a testbed with 33.2 GB working malware collected from VirusTotal. Despite the sub-optimal nature of our method, it was able to surprisingly enhance the robustness of three known open-source ML-based malware detectors (LGBM, MalConv, and NonNeg) against the adversarial malware variants by 4, 7, and 11 times, respectively.
AB - Empowered by the recent development in Ma-chine Learning (ML), signatureless ML-based malware detectors present promising performance in identifying unseen mal ware variants and zero days without requiring expensive dynamic malware analysis. However, it has been recently shown that ML-based malware detectors are vulnerable to adversarial malware attacks, in which an attacker modifies a known malware exe-cutable to trick the malware detector into recognizing the modi-fied variant as benign. Adversarial malware example generation has become an emerging area in adversarial ML that studies creating functionality-preserving adversarial malware variants. Advancements in this area have led to an eternal game between the adversary and defender. While the area has attracted much attention in the security community, a large body of these studies merely focuses on attack methods against ML-based malware detectors. There has been little work on understanding how these adversarial variants can be systematically used by the defender to strengthen the robustness of these detectors and stand ahead of the adversary. Latest efforts have led to emergence of adversarial learning. In this work, we propose a simple wargame approach to empirically conduct the adversarial minimax optimization underlying in the adversarial learning for improving the robustness of ML-based malware detectors. Our proposed approach employs adversarial malware variants generated from a reinforcement learning-based adversarial attack policy in a minimax game alternating between strengthening the attack policy and improving the detectors' robustness. We evaluated the effectiveness of our approach on a testbed with 33.2 GB working malware collected from VirusTotal. Despite the sub-optimal nature of our method, it was able to surprisingly enhance the robustness of three known open-source ML-based malware detectors (LGBM, MalConv, and NonNeg) against the adversarial malware variants by 4, 7, and 11 times, respectively.
KW - adversarial learning
KW - adversarial malware variants
KW - adversarial minimax game
KW - adversarial robustness
KW - machine learning-based malware detection
UR - http://www.scopus.com/inward/record.url?scp=85148436341&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85148436341&partnerID=8YFLogxK
U2 - 10.1109/ICDMW58026.2022.00079
DO - 10.1109/ICDMW58026.2022.00079
M3 - Conference contribution
AN - SCOPUS:85148436341
T3 - IEEE International Conference on Data Mining Workshops, ICDMW
SP - 567
EP - 576
BT - Proceedings - 22nd IEEE International Conference on Data Mining Workshops, ICDMW 2022
A2 - Candan, K. Selcuk
A2 - Dinh, Thang N.
A2 - Thai, My T.
A2 - Washio, Takashi
PB - IEEE Computer Society
T2 - 22nd IEEE International Conference on Data Mining Workshops, ICDMW 2022
Y2 - 28 November 2022 through 1 December 2022
ER -