Evading Deep Learning-Based Malware Detectors via Obfuscation: A Deep Reinforcement Learning Approach

Brian Etter; James Lee Hu; Mohammadreza Ebrahimi; Weifeng Li; Xin Li; Hsinchun Chen

doi:10.1109/ICDMW60847.2023.00169

Evading Deep Learning-Based Malware Detectors via Obfuscation: A Deep Reinforcement Learning Approach

Brian Etter, James Lee Hu, Mohammadreza Ebrahimi, Weifeng Li, Xin Li, Hsinchun Chen

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Adversarial Malware Generation (AMG), the generation of adversarial malware variants to strengthen Deep Learning (DL)-based malware detectors has emerged as a crucial tool in the development of proactive cyberdefense. However, the majority of extant works offer subtle perturbations or additions to executable files and do not explore full-file obfuscation. In this study, we show that an open-source encryption tool coupled with a Reinforcement Learning (RL) framework can successfully obfuscate malware to evade state-of-the-art malware detection engines and outperform techniques that use advanced modification methods. Our results show that the proposed method improves the evasion rate from 27%-49% compared to widely-used state-of-the-art reinforcement learning-based methods.

Original language	English (US)
Title of host publication	Proceedings - 23rd IEEE International Conference on Data Mining Workshops, ICDMW 2023
Editors	Jihe Wang, Yi He, Thang N. Dinh, Christan Grant, Meikang Qiu, Witold Pedrycz
Publisher	IEEE Computer Society
Pages	1313-1321
Number of pages	9
ISBN (Electronic)	9798350381641
DOIs	https://doi.org/10.1109/ICDMW60847.2023.00169
State	Published - 2023
Event	23rd IEEE International Conference on Data Mining Workshops, ICDMW 2023 - Shanghai, China Duration: Dec 1 2023 → Dec 4 2023

Publication series

Name	IEEE International Conference on Data Mining Workshops, ICDMW
ISSN (Print)	2375-9232
ISSN (Electronic)	2375-9259

Conference

Conference	23rd IEEE International Conference on Data Mining Workshops, ICDMW 2023
Country/Territory	China
City	Shanghai
Period	12/1/23 → 12/4/23

Keywords

Adversarial Malware Generation
Adversarial Malware Variants
Adversarial Robustness
Obfuscation
Reinforcement Learning

ASJC Scopus subject areas

Computer Science Applications
Software

Access to Document

10.1109/ICDMW60847.2023.00169

Cite this

Etter, B., Hu, J. L., Ebrahimi, M., Li, W., Li, X., & Chen, H. (2023). Evading Deep Learning-Based Malware Detectors via Obfuscation: A Deep Reinforcement Learning Approach. In J. Wang, Y. He, T. N. Dinh, C. Grant, M. Qiu, & W. Pedrycz (Eds.), Proceedings - 23rd IEEE International Conference on Data Mining Workshops, ICDMW 2023 (pp. 1313-1321). (IEEE International Conference on Data Mining Workshops, ICDMW). IEEE Computer Society. https://doi.org/10.1109/ICDMW60847.2023.00169

Evading Deep Learning-Based Malware Detectors via Obfuscation: A Deep Reinforcement Learning Approach. / Etter, Brian; Hu, James Lee; Ebrahimi, Mohammadreza et al.
Proceedings - 23rd IEEE International Conference on Data Mining Workshops, ICDMW 2023. ed. / Jihe Wang; Yi He; Thang N. Dinh; Christan Grant; Meikang Qiu; Witold Pedrycz. IEEE Computer Society, 2023. p. 1313-1321 (IEEE International Conference on Data Mining Workshops, ICDMW).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Etter, B, Hu, JL, Ebrahimi, M, Li, W, Li, X & Chen, H 2023, Evading Deep Learning-Based Malware Detectors via Obfuscation: A Deep Reinforcement Learning Approach. in J Wang, Y He, TN Dinh, C Grant, M Qiu & W Pedrycz (eds), Proceedings - 23rd IEEE International Conference on Data Mining Workshops, ICDMW 2023. IEEE International Conference on Data Mining Workshops, ICDMW, IEEE Computer Society, pp. 1313-1321, 23rd IEEE International Conference on Data Mining Workshops, ICDMW 2023, Shanghai, China, 12/1/23. https://doi.org/10.1109/ICDMW60847.2023.00169

Etter B, Hu JL, Ebrahimi M, Li W, Li X, Chen H. Evading Deep Learning-Based Malware Detectors via Obfuscation: A Deep Reinforcement Learning Approach. In Wang J, He Y, Dinh TN, Grant C, Qiu M, Pedrycz W, editors, Proceedings - 23rd IEEE International Conference on Data Mining Workshops, ICDMW 2023. IEEE Computer Society. 2023. p. 1313-1321. (IEEE International Conference on Data Mining Workshops, ICDMW). doi: 10.1109/ICDMW60847.2023.00169

Etter, Brian ; Hu, James Lee ; Ebrahimi, Mohammadreza et al. / Evading Deep Learning-Based Malware Detectors via Obfuscation : A Deep Reinforcement Learning Approach. Proceedings - 23rd IEEE International Conference on Data Mining Workshops, ICDMW 2023. editor / Jihe Wang ; Yi He ; Thang N. Dinh ; Christan Grant ; Meikang Qiu ; Witold Pedrycz. IEEE Computer Society, 2023. pp. 1313-1321 (IEEE International Conference on Data Mining Workshops, ICDMW).

@inproceedings{974c5185523e4615adb5e60052dda466,

title = "Evading Deep Learning-Based Malware Detectors via Obfuscation: A Deep Reinforcement Learning Approach",

abstract = "Adversarial Malware Generation (AMG), the generation of adversarial malware variants to strengthen Deep Learning (DL)-based malware detectors has emerged as a crucial tool in the development of proactive cyberdefense. However, the majority of extant works offer subtle perturbations or additions to executable files and do not explore full-file obfuscation. In this study, we show that an open-source encryption tool coupled with a Reinforcement Learning (RL) framework can successfully obfuscate malware to evade state-of-the-art malware detection engines and outperform techniques that use advanced modification methods. Our results show that the proposed method improves the evasion rate from 27%-49% compared to widely-used state-of-the-art reinforcement learning-based methods.",

keywords = "Adversarial Malware Generation, Adversarial Malware Variants, Adversarial Robustness, Obfuscation, Reinforcement Learning",

author = "Brian Etter and Hu, {James Lee} and Mohammadreza Ebrahimi and Weifeng Li and Xin Li and Hsinchun Chen",

note = "Publisher Copyright: {\textcopyright} 2023 IEEE.; 23rd IEEE International Conference on Data Mining Workshops, ICDMW 2023 ; Conference date: 01-12-2023 Through 04-12-2023",

year = "2023",

doi = "10.1109/ICDMW60847.2023.00169",

language = "English (US)",

series = "IEEE International Conference on Data Mining Workshops, ICDMW",

publisher = "IEEE Computer Society",

pages = "1313--1321",

editor = "Jihe Wang and Yi He and Dinh, {Thang N.} and Christan Grant and Meikang Qiu and Witold Pedrycz",

booktitle = "Proceedings - 23rd IEEE International Conference on Data Mining Workshops, ICDMW 2023",

}

TY - GEN

T1 - Evading Deep Learning-Based Malware Detectors via Obfuscation

T2 - 23rd IEEE International Conference on Data Mining Workshops, ICDMW 2023

AU - Etter, Brian

AU - Hu, James Lee

AU - Ebrahimi, Mohammadreza

AU - Li, Weifeng

AU - Li, Xin

AU - Chen, Hsinchun

PY - 2023

Y1 - 2023

N2 - Adversarial Malware Generation (AMG), the generation of adversarial malware variants to strengthen Deep Learning (DL)-based malware detectors has emerged as a crucial tool in the development of proactive cyberdefense. However, the majority of extant works offer subtle perturbations or additions to executable files and do not explore full-file obfuscation. In this study, we show that an open-source encryption tool coupled with a Reinforcement Learning (RL) framework can successfully obfuscate malware to evade state-of-the-art malware detection engines and outperform techniques that use advanced modification methods. Our results show that the proposed method improves the evasion rate from 27%-49% compared to widely-used state-of-the-art reinforcement learning-based methods.

AB - Adversarial Malware Generation (AMG), the generation of adversarial malware variants to strengthen Deep Learning (DL)-based malware detectors has emerged as a crucial tool in the development of proactive cyberdefense. However, the majority of extant works offer subtle perturbations or additions to executable files and do not explore full-file obfuscation. In this study, we show that an open-source encryption tool coupled with a Reinforcement Learning (RL) framework can successfully obfuscate malware to evade state-of-the-art malware detection engines and outperform techniques that use advanced modification methods. Our results show that the proposed method improves the evasion rate from 27%-49% compared to widely-used state-of-the-art reinforcement learning-based methods.

KW - Adversarial Malware Generation

KW - Adversarial Malware Variants

KW - Adversarial Robustness

KW - Obfuscation

KW - Reinforcement Learning

UR - http://www.scopus.com/inward/record.url?scp=85186142753&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85186142753&partnerID=8YFLogxK

U2 - 10.1109/ICDMW60847.2023.00169

DO - 10.1109/ICDMW60847.2023.00169

M3 - Conference contribution

AN - SCOPUS:85186142753

T3 - IEEE International Conference on Data Mining Workshops, ICDMW

SP - 1313

EP - 1321

BT - Proceedings - 23rd IEEE International Conference on Data Mining Workshops, ICDMW 2023

A2 - Wang, Jihe

A2 - He, Yi

A2 - Dinh, Thang N.

A2 - Grant, Christan

A2 - Qiu, Meikang

A2 - Pedrycz, Witold

PB - IEEE Computer Society

Y2 - 1 December 2023 through 4 December 2023

ER -

Evading Deep Learning-Based Malware Detectors via Obfuscation: A Deep Reinforcement Learning Approach

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this