Empirical evaluation of hyper-parameter optimization techniques for deep learning-based malware detectors

• Main Authors: SHAR, Lwin Khin, TA, Nguyen Binh Duong, YEO, Yao Cong, FAN, Jiani
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2024
• Subjects: Malware detection; Hyper-parameter tuning; Bayesian optimization; Deep learning; Information Security
• Online Access: https://ink.library.smu.edu.sg/sis_research/9797
• Institution: Singapore Management University

In machine learning, hyper-parameter optimization (HPO) aims to tune the set of parameters that controls the learning process. HPO could be time-consuming and resource-intensive due to the huge parameter search space and the complexity of models such as deep neural networks. Many of the existing HPO techniques tend to be variants of Bayesian optimization methods; each of which has been applied successfully for model tuning in different application domains. However, these Bayesian optimization methods have not been systematically evaluated against each other in the context of deep learning based malware detection. In this paper, we report a large-scale empirical study comparing popular HPO techniques on the performance of deep learning based malware classifiers. We use a diverse collection of seven datasets covering the most typical features used in malware detection. We conduct our experiments with Ray Tune, a distributed tuning platform, and popular optimization libraries such as Optuna, HyperOpt, Nevergrad, etc., across a wide range of computing platforms including AWS EC2, high-performance workstation, and laptop computers. Our extensive experiments provide useful insights into the application of different HPO techniques in deep learning based malware detection.