An Adversarial Reinforcement Learning Framework for Robust Machine Learning-based Malware Detection

Mohammadreza Reza Ebrahimi, Weifeng Li, Yidong Chai, Jason Pacheco, Hsinchun Chen

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Scopus citations

Abstract

Empowered by the recent development in Ma-chine Learning (ML), signatureless ML-based malware detectors present promising performance in identifying unseen mal ware variants and zero days without requiring expensive dynamic malware analysis. However, it has been recently shown that ML-based malware detectors are vulnerable to adversarial malware attacks, in which an attacker modifies a known malware exe-cutable to trick the malware detector into recognizing the modi-fied variant as benign. Adversarial malware example generation has become an emerging area in adversarial ML that studies creating functionality-preserving adversarial malware variants. Advancements in this area have led to an eternal game between the adversary and defender. While the area has attracted much attention in the security community, a large body of these studies merely focuses on attack methods against ML-based malware detectors. There has been little work on understanding how these adversarial variants can be systematically used by the defender to strengthen the robustness of these detectors and stand ahead of the adversary. Latest efforts have led to emergence of adversarial learning. In this work, we propose a simple wargame approach to empirically conduct the adversarial minimax optimization underlying in the adversarial learning for improving the robustness of ML-based malware detectors. Our proposed approach employs adversarial malware variants generated from a reinforcement learning-based adversarial attack policy in a minimax game alternating between strengthening the attack policy and improving the detectors' robustness. We evaluated the effectiveness of our approach on a testbed with 33.2 GB working malware collected from VirusTotal. Despite the sub-optimal nature of our method, it was able to surprisingly enhance the robustness of three known open-source ML-based malware detectors (LGBM, MalConv, and NonNeg) against the adversarial malware variants by 4, 7, and 11 times, respectively.

Original languageEnglish (US)
Title of host publicationProceedings - 22nd IEEE International Conference on Data Mining Workshops, ICDMW 2022
EditorsK. Selcuk Candan, Thang N. Dinh, My T. Thai, Takashi Washio
PublisherIEEE Computer Society
Pages567-576
Number of pages10
ISBN (Electronic)9798350346091
DOIs
StatePublished - 2022
Event22nd IEEE International Conference on Data Mining Workshops, ICDMW 2022 - Orlando, United States
Duration: Nov 28 2022Dec 1 2022

Publication series

NameIEEE International Conference on Data Mining Workshops, ICDMW
Volume2022-November
ISSN (Print)2375-9232
ISSN (Electronic)2375-9259

Conference

Conference22nd IEEE International Conference on Data Mining Workshops, ICDMW 2022
Country/TerritoryUnited States
CityOrlando
Period11/28/2212/1/22

Keywords

  • adversarial learning
  • adversarial malware variants
  • adversarial minimax game
  • adversarial robustness
  • machine learning-based malware detection

ASJC Scopus subject areas

  • Computer Science Applications
  • Software

Fingerprint

Dive into the research topics of 'An Adversarial Reinforcement Learning Framework for Robust Machine Learning-based Malware Detection'. Together they form a unique fingerprint.

Cite this