Novel techniques in recovering, embedding, and enforcing policies for control-flow integrity

• Main Author: LIN, Yan
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2021
• Subjects: Control-Flow Integrity; Control-Flow Hijacking Attack; Function Signature; Deep Learning; Dynamic Code Optimization; Secret Sharing; Databases and Information Systems; Information Security
• Online Access: https://ink.library.smu.edu.sg/etd_coll/318; https://ink.library.smu.edu.sg/cgi/viewcontent.cgi?article=1318&context=etd_coll
• Institution: Singapore Management University

Control-Flow Integrity (CFI) is an attractive security property with which most injected and code-reuse attacks can be defeated, including advanced attacking techniques like Return-Oriented Programming (ROP). CFI extracts a control-flow graph (CFG) for a given program and instruments the program to respect the CFG. Specifically, checks are inserted before indirect branch instructions. Before these instructions are executed during runtime, the checks consult the CFG to ensure that the indirect branch is allowed to reach the intended target. Hence, any sort of controlflow hijacking would be prevented. There are three fundamental components in CFI enforcement. The first component is accurately recovering the policy (CFG). Usually, the more precise the policy (CFG) is, the more security CFI improves, but precise CFG generation was considered hard without the support of source code. The second one is embedding the CFI policy securely. Current CFI enforcement usually inserts checks before indirect branches to consult a read-only table which stores the valid CFG information. However, this kind of read-only table can be overwritten by some kinds of attacks(e.g., Rowhammer attack and data-oriented programming). The third component is to efficiently enforce the CFI policy. In current approaches, no matter whether there are attacks, the CFI checks are always executed whenever there is an indirect control-flow transfer. Therefore, it is critical to minimize the performance impactof the CFI checks. In this dissertation, we propose novel solutions to handle these three fundamental components. We systematically study how compiler optimization would impact CFG recovery by investigating two methods that recover CFI policy based on function signature matching at the binary level and propose our novel improved mechanism to more accurately recover function signature. We also propose an enhanceddeep learning approach to recover function signature by including domain-specific knowledge to the dataset. To embed CFI policy securely, we design a novel platform which encodes the policy into the machine instructions directly without relying on consulting any read-only data structure by making use of the idea of instruction-set randomization. In it, each basic block is encrypted with a key derived from the CFG. To efficiently enforce CFI policy, we make use of a mature dynamic code optimization platform called DynamoRIO to enforce the policy so that it only requires to do the CFI check when needed.