Detection of buffer overflow vulnerabilities in C/C++ with pattern based limited symbolic evaluation

• Main Authors: Ding, Sun, Tan, Hee Beng Kuan, Liu, Kaiping, Chandramohan, Mahinthan, Zhang, Hongyu
• Other Authors: School of Electrical and Electronic Engineering
• Format: Conference or Workshop Item
• Language: English
• Published: 2013
• Subjects: DRNTU::Engineering::Electrical and electronic engineering
• Online Access: https://hdl.handle.net/10356/99418; http://hdl.handle.net/10220/13011
• Institution: Nanyang Technological University

Buffer overflow vulnerability is one of the major security threats for applications written in C/C++. Among the existing approaches for detecting buffer overflow vulnerability, though flow sensitive based approaches offer higher precision but they are limited by heavy overhead and the fact that many constraints are unsolvable. We propose a novel method to efficiently detect vulnerable buffer overflows in any given control flow graph through recognizing two patterns. The proposed approach first uses syntax analysis to filter away those branches that cannot possibly comply with any of the two patterns before applying a limited symbolic evaluation for a precise matching against the patterns. The proposed approach only needs to evaluate a limited set of selected branch predicates according to the patterns and avoids the need to deal with a large number of general branch predicates. This significantly improves the scalability while not sacrificing the detection precision. Our experiments demonstrate the scalability and efficiency of the proposed method, which demonstrates its applicability.