ReSIL: Revivifying function signature inference using deep learning with domain-specific knowledge

• Main Authors: LIN, Yan, GAO, Debin, LO, David
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2022
• Subjects: Function Signature; Recurrent Neural Network; Compiler Optimization; Information Security; OS and Networks
• Online Access: https://ink.library.smu.edu.sg/sis_research/7355; https://ink.library.smu.edu.sg/context/sis_research/article/8358/viewcontent/codaspy_22.pdf
• Institution: Singapore Management University

Function signature recovery is important for binary analysis and security enhancement, such as bug finding and control-flow integrity enforcement. However, binary executables typically have crucial information vital for function signature recovery stripped off during compilation. To make things worse, recent studies show that many compiler optimization strategies further complicate the recovery of function signatures with intended violations to function calling conventions.In this paper, we first perform a systematic study to quantify the extent to which compiler optimizations (negatively) impact the accuracy of existing deep learning techniques for function signature recovery. Our experiments show that a state-of-the-art deep learning technique has its accuracy dropped from 98.7% to 87.7% when training and testing optimized binaries. We further identify specific weaknesses in existing approaches and propose an enhanced deep learning approach named \sysname (\underlineRe vivifying Function \underlineS ignature \underlineI nference using Deep \underlineL earning) to incorporate compiler-optimization-specific domain knowledge into the learning process. Our experimental results show that \sysname significantly improves the accuracy and F1 score in inferring function signatures, e.g., with accuracy in inferring the number of arguments for callees compiled with optimization flag O1 from 84.8% to 92.67%. We also demonstrate security implications of \sysname in Control-Flow Integrity enforcement in stopping potential Counterfeit Object-Oriented Programming (COOP) attacks.