Active Malware Analysis using Stochastic Games

• Main Authors: WILLIAMSON, Simon, VARAKANTHAM, Pradeep Reddy, GAO, Debin, ONG, Chen Hui
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2012
• Subjects: Malware Analysis; Stochastic Game; Autonomous agents; Computer crime; Game theory; Intrusion detection; Multi agent systems; Network security; Artificial Intelligence and Robotics; Information Security
• Online Access: https://ink.library.smu.edu.sg/sis_research/1476; https://ink.library.smu.edu.sg/context/sis_research/article/2475/viewcontent/ActiveMalwareAnalysiStochastic_2012.pdf
• Institution: Singapore Management University

Cyber security is increasingly important for defending computer systems from loss of privacy or unauthorised use. One important aspect is threat analysis - how does an attacker infiltrate a system and what do they want once they are inside. This paper considers the problem of Active Malware Analysis, where we learn about the human or software intruder by actively interacting with it with the goal of learning about its behaviours and intentions, whilst at the same time that intruder may be trying to avoid detection or showing those behaviours and intentions. This game-theoretic active learning is then used to obtain a behavioural clustering of malware, an important contribution for both understanding malware at a high level and more crucially, for the deployment of effective anti-malware defences. This paper makes the following contributions: (i) A formal definition of the game-theoretic active malware analysis problem; (ii) A fast algorithm for learning about a malware in the active analysis problem which utilises the concept of reducing entropy in the beliefs about the malware; (iii) A virtual machine based agent architecture for the implementation of the active malware analysis problem and (iv) A behaviour based clustering of malware behaviour which is shown to be more accurate than a similar clustering using only passive information about the malware.