Detection and classification of malicious JavaScript via attack behavior modelling

• Main Authors: XUE, Yinxing, WANG, Junjie, LIU, Yang, XIAO, Hao, SUN, Jun, CHANDRAMOHAN, Mahinthan
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2015
• Subjects: malware detection; malicious JavaScript; L*; behavior modelling; Programming Languages and Compilers; Software Engineering
• Online Access: https://ink.library.smu.edu.sg/sis_research/4953; https://ink.library.smu.edu.sg/context/sis_research/article/5956/viewcontent/issta2015.pdf
• Institution: Singapore Management University

Existing malicious JavaScript (JS) detection tools and commercial anti-virus tools mostly use feature-based or signature-based approaches to detect JS malware. These tools are weak in resistance to obfuscation and JS malware variants, not mentioning about providing detailed information of attack behaviors. Such limitations root in the incapability of capturing attack behaviors in these approches. In this paper, we propose to use Deterministic Finite Automaton (DFA) to abstract and summarize common behaviors of malicious JS of the same attack type. We propose an automatic behavior learning framework, named JS∗ , to learn DFA from dynamic execution traces of JS malware, where we implement an effective online teacher by combining data dependency analysis, defense rules and trace replay mechanism. We evaluate JS∗ using real world data of 10000 benign and 276 malicious JS samples to cover 8 most-infectious attack types. The results demonstrate the scalability and effectiveness of our approach in the malware detection and classification, compared with commercial anti-virus tools. We also show how to use our DFAs to detect variants and new attacks.