Catch you with cache: Out-of-VM introspection to trace malicious executions

• Main Authors: SU, Chao, DING, Xuhua, ZENG, Qinghai
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2021
• Subjects: Out-of-VM Introspection; Cache; Malware Analysis; Non-intrusiveness; Transparency; Databases and Information Systems; Information Security
• Online Access: https://ink.library.smu.edu.sg/sis_research/6737; https://ink.library.smu.edu.sg/context/sis_research/article/7740/viewcontent/357200a326.pdf
• Institution: Singapore Management University

Out-of-VM introspection is an imperative part of security analysis. The legacy methods either modify the system, introducing enormous overhead, or rely heavily on hardware features, which are neither available nor practical in most cloud environments. In this paper, we propose a novel analysis method, named as Catcher, that utilizes CPU cache to perform out-of-VM introspection. Catcher does not make any modifications to the target program and its running environment, nor demands special hardware support. Implemented upon Linux KVM, it natively introspects the target's virtual memory. More importantly, it uses the cache-based side channel to infer the target control flow. To deal with the inherent limitations of the side channel, we propose several heuristics to improve the accuracy and stability of Catcher. Our experiments against various malware armored with packing techniques show that Catcher can recover the control flow in real time with around 67% to 97% accuracy scores. Catcher incurs a negligible overhead to the system and can be launched at anytime to monitor an ongoing attack inside a virtual machine.