DynOpVm: VM-based software obfuscation with dynamic opcode mapping

• Main Authors: CHENG, Xiaoyang, LIN, Yan, GAO, Debin
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2019
• Subjects: Frequency analysis; Software obfuscation; Virtualization; Information Security
• Online Access: https://ink.library.smu.edu.sg/sis_research/4687; https://ink.library.smu.edu.sg/context/sis_research/article/5690/viewcontent/DynOpVm_acns19_av.pdf
• Institution: Singapore Management University

VM-based software obfuscation has emerged as an effective technique for program obfuscation. Despite various attempts in improving its effectiveness and security, existing VM-based software obfuscators use potentially multiple but static secret mappings between virtual and native opcodes to hide the underlying instructions. In this paper, we present an attack using frequency analysis to effectively recover the secret mapping to compromise the protection, and then propose a novel VM-based obfuscator in which each basic block uses a dynamic and control-flow-aware mapping between the virtual and native instructions. We show that our proposed VM-based obfuscator not only renders the frequency analysis attack ineffective, but dictates the execution and program analysis to follow the original control flow of the program, making state-of-the-art backward tainting and slicing ineffective. We implement a prototype of our VM-based obfuscator and show its effectiveness with experiments on SPEC benchmarking and other real-world applications.