Catch you with cache: Out-of-VM introspection to trace malicious executions

Out-of-VM introspection is an imperative part of security analysis. The legacy methods either modify the system, introducing enormous overhead, or rely heavily on hardware features, which are neither available nor practical in most cloud environments. In this paper, we propose a novel analysis metho...

Full description

Saved in:
Bibliographic Details
Main Authors: SU, Chao, DING, Xuhua, ZENG, Qinghai
Format: text
Language:English
Published: Institutional Knowledge at Singapore Management University 2021
Subjects:
Online Access:https://ink.library.smu.edu.sg/sis_research/6737
https://ink.library.smu.edu.sg/context/sis_research/article/7740/viewcontent/357200a326.pdf
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Singapore Management University
Language: English
Description
Summary:Out-of-VM introspection is an imperative part of security analysis. The legacy methods either modify the system, introducing enormous overhead, or rely heavily on hardware features, which are neither available nor practical in most cloud environments. In this paper, we propose a novel analysis method, named as Catcher, that utilizes CPU cache to perform out-of-VM introspection. Catcher does not make any modifications to the target program and its running environment, nor demands special hardware support. Implemented upon Linux KVM, it natively introspects the target's virtual memory. More importantly, it uses the cache-based side channel to infer the target control flow. To deal with the inherent limitations of the side channel, we propose several heuristics to improve the accuracy and stability of Catcher. Our experiments against various malware armored with packing techniques show that Catcher can recover the control flow in real time with around 67% to 97% accuracy scores. Catcher incurs a negligible overhead to the system and can be launched at anytime to monitor an ongoing attack inside a virtual machine.