Evading Deep Learning-Based Malware Detectors via Obfuscation: A Deep Reinforcement Learning Approach

Brian Etter; James Lee Hu; Mohammadreza Ebrahimi; Weifeng Li; Xin Li; Hsinchun Chen

doi:10.1109/ICDM58522.2023.00019

Evading Deep Learning-Based Malware Detectors via Obfuscation: A Deep Reinforcement Learning Approach

Brian Etter, James Lee Hu, Mohammadreza Ebrahimi, Weifeng Li, Xin Li, Hsinchun Chen

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Adversarial Malware Generation (AMG), the generation of adversarial malware variants to strengthen Deep Learning (DL)-based malware detectors has emerged as a crucial tool in the development of proactive cyberdefense. However, the majority of extant works offer subtle perturbations or additions to executable files and do not explore full-file obfuscation. In this study, we show that an open-source encryption tool coupled with a Reinforcement Learning (RL) framework can successfully obfuscate malware to evade state-of-the-art malware detection engines and outperform techniques that use advanced modification methods. Our results show that the proposed method improves the evasion rate from 27%-49% compared to widely-used state-of-the-art reinforcement learning-based methods.

Original language	English (US)
Title of host publication	Proceedings - 23rd IEEE International Conference on Data Mining, ICDM 2023
Editors	Guihai Chen, Latifur Khan, Xiaofeng Gao, Meikang Qiu, Witold Pedrycz, Xindong Wu
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	101-109
Number of pages	9
ISBN (Electronic)	9798350307887
DOIs	https://doi.org/10.1109/ICDM58522.2023.00019
State	Published - 2023
Event	23rd IEEE International Conference on Data Mining, ICDM 2023 - Shanghai, China Duration: Dec 1 2023 → Dec 4 2023

Publication series

Name	Proceedings - IEEE International Conference on Data Mining, ICDM
ISSN (Print)	1550-4786

Conference

Conference	23rd IEEE International Conference on Data Mining, ICDM 2023
Country/Territory	China
City	Shanghai
Period	12/1/23 → 12/4/23

Keywords

Adversarial Malware Generation
Adversarial Malware Variants
Adversarial Robustness
Obfuscation
Reinforcement Learning

ASJC Scopus subject areas

General Engineering

Access to Document

10.1109/ICDM58522.2023.00019

Cite this

Etter, B., Hu, J. L., Ebrahimi, M., Li, W., Li, X., & Chen, H. (2023). Evading Deep Learning-Based Malware Detectors via Obfuscation: A Deep Reinforcement Learning Approach. In G. Chen, L. Khan, X. Gao, M. Qiu, W. Pedrycz, & X. Wu (Eds.), Proceedings - 23rd IEEE International Conference on Data Mining, ICDM 2023 (pp. 101-109). (Proceedings - IEEE International Conference on Data Mining, ICDM). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICDM58522.2023.00019

Evading Deep Learning-Based Malware Detectors via Obfuscation: A Deep Reinforcement Learning Approach. / Etter, Brian; Hu, James Lee; Ebrahimi, Mohammadreza et al.
Proceedings - 23rd IEEE International Conference on Data Mining, ICDM 2023. ed. / Guihai Chen; Latifur Khan; Xiaofeng Gao; Meikang Qiu; Witold Pedrycz; Xindong Wu. Institute of Electrical and Electronics Engineers Inc., 2023. p. 101-109 (Proceedings - IEEE International Conference on Data Mining, ICDM).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Etter, B, Hu, JL, Ebrahimi, M, Li, W, Li, X & Chen, H 2023, Evading Deep Learning-Based Malware Detectors via Obfuscation: A Deep Reinforcement Learning Approach. in G Chen, L Khan, X Gao, M Qiu, W Pedrycz & X Wu (eds), Proceedings - 23rd IEEE International Conference on Data Mining, ICDM 2023. Proceedings - IEEE International Conference on Data Mining, ICDM, Institute of Electrical and Electronics Engineers Inc., pp. 101-109, 23rd IEEE International Conference on Data Mining, ICDM 2023, Shanghai, China, 12/1/23. https://doi.org/10.1109/ICDM58522.2023.00019

Etter B, Hu JL, Ebrahimi M, Li W, Li X, Chen H. Evading Deep Learning-Based Malware Detectors via Obfuscation: A Deep Reinforcement Learning Approach. In Chen G, Khan L, Gao X, Qiu M, Pedrycz W, Wu X, editors, Proceedings - 23rd IEEE International Conference on Data Mining, ICDM 2023. Institute of Electrical and Electronics Engineers Inc. 2023. p. 101-109. (Proceedings - IEEE International Conference on Data Mining, ICDM). doi: 10.1109/ICDM58522.2023.00019

Etter, Brian ; Hu, James Lee ; Ebrahimi, Mohammadreza et al. / Evading Deep Learning-Based Malware Detectors via Obfuscation : A Deep Reinforcement Learning Approach. Proceedings - 23rd IEEE International Conference on Data Mining, ICDM 2023. editor / Guihai Chen ; Latifur Khan ; Xiaofeng Gao ; Meikang Qiu ; Witold Pedrycz ; Xindong Wu. Institute of Electrical and Electronics Engineers Inc., 2023. pp. 101-109 (Proceedings - IEEE International Conference on Data Mining, ICDM).

@inproceedings{baa5fa3901fd4f2bb2d0b168f2278798,

title = "Evading Deep Learning-Based Malware Detectors via Obfuscation: A Deep Reinforcement Learning Approach",

abstract = "Adversarial Malware Generation (AMG), the generation of adversarial malware variants to strengthen Deep Learning (DL)-based malware detectors has emerged as a crucial tool in the development of proactive cyberdefense. However, the majority of extant works offer subtle perturbations or additions to executable files and do not explore full-file obfuscation. In this study, we show that an open-source encryption tool coupled with a Reinforcement Learning (RL) framework can successfully obfuscate malware to evade state-of-the-art malware detection engines and outperform techniques that use advanced modification methods. Our results show that the proposed method improves the evasion rate from 27%-49% compared to widely-used state-of-the-art reinforcement learning-based methods.",

keywords = "Adversarial Malware Generation, Adversarial Malware Variants, Adversarial Robustness, Obfuscation, Reinforcement Learning",

author = "Brian Etter and Hu, {James Lee} and Mohammadreza Ebrahimi and Weifeng Li and Xin Li and Hsinchun Chen",

note = "Publisher Copyright: {\textcopyright} 2023 IEEE.; 23rd IEEE International Conference on Data Mining, ICDM 2023 ; Conference date: 01-12-2023 Through 04-12-2023",

year = "2023",

doi = "10.1109/ICDM58522.2023.00019",

language = "English (US)",

series = "Proceedings - IEEE International Conference on Data Mining, ICDM",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "101--109",

editor = "Guihai Chen and Latifur Khan and Xiaofeng Gao and Meikang Qiu and Witold Pedrycz and Xindong Wu",

booktitle = "Proceedings - 23rd IEEE International Conference on Data Mining, ICDM 2023",

}

TY - GEN

T1 - Evading Deep Learning-Based Malware Detectors via Obfuscation

T2 - 23rd IEEE International Conference on Data Mining, ICDM 2023

AU - Etter, Brian

AU - Hu, James Lee

AU - Ebrahimi, Mohammadreza

AU - Li, Weifeng

AU - Li, Xin

AU - Chen, Hsinchun

PY - 2023

Y1 - 2023

N2 - Adversarial Malware Generation (AMG), the generation of adversarial malware variants to strengthen Deep Learning (DL)-based malware detectors has emerged as a crucial tool in the development of proactive cyberdefense. However, the majority of extant works offer subtle perturbations or additions to executable files and do not explore full-file obfuscation. In this study, we show that an open-source encryption tool coupled with a Reinforcement Learning (RL) framework can successfully obfuscate malware to evade state-of-the-art malware detection engines and outperform techniques that use advanced modification methods. Our results show that the proposed method improves the evasion rate from 27%-49% compared to widely-used state-of-the-art reinforcement learning-based methods.

AB - Adversarial Malware Generation (AMG), the generation of adversarial malware variants to strengthen Deep Learning (DL)-based malware detectors has emerged as a crucial tool in the development of proactive cyberdefense. However, the majority of extant works offer subtle perturbations or additions to executable files and do not explore full-file obfuscation. In this study, we show that an open-source encryption tool coupled with a Reinforcement Learning (RL) framework can successfully obfuscate malware to evade state-of-the-art malware detection engines and outperform techniques that use advanced modification methods. Our results show that the proposed method improves the evasion rate from 27%-49% compared to widely-used state-of-the-art reinforcement learning-based methods.

KW - Adversarial Malware Generation

KW - Adversarial Malware Variants

KW - Adversarial Robustness

KW - Obfuscation

KW - Reinforcement Learning

UR - http://www.scopus.com/inward/record.url?scp=85185404617&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85185404617&partnerID=8YFLogxK

U2 - 10.1109/ICDM58522.2023.00019

DO - 10.1109/ICDM58522.2023.00019

M3 - Conference contribution

AN - SCOPUS:85185404617

T3 - Proceedings - IEEE International Conference on Data Mining, ICDM

SP - 101

EP - 109

BT - Proceedings - 23rd IEEE International Conference on Data Mining, ICDM 2023

A2 - Chen, Guihai

A2 - Khan, Latifur

A2 - Gao, Xiaofeng

A2 - Qiu, Meikang

A2 - Pedrycz, Witold

A2 - Wu, Xindong

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 1 December 2023 through 4 December 2023

ER -

Evading Deep Learning-Based Malware Detectors via Obfuscation: A Deep Reinforcement Learning Approach

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this