Be sensitive and collaborative: Analyzing impact of coverage metrics in Greybox fuzzing

• Main Authors: WANG, Jinghan, DUAN, Yue, SONG, Wei, YIN, Heng, SONG, Chengyu
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2019
• Subjects: Computation power; Computing resource; Concolic execution; Coverage metrics; Real-world; Software bug; Systematic study; Vulnerability finding; Information Security
• Online Access: https://ink.library.smu.edu.sg/sis_research/8169; https://ink.library.smu.edu.sg/context/sis_research/article/9172/viewcontent/Be_Sensitive_and_Collaborative_Analyzing_Impact_of_Coverage_Metrics_in_Greybox_Fuzzing.pdf
• Institution: Singapore Management University

Coverage-guided greybox fuzzing has become one of the most common techniques for finding software bugs. Coverage metric, which decides how a fuzzer selects new seeds, is an essential parameter of fuzzing and can significantly affect the results. While there are many existing works on the effectiveness of different coverage metrics on software testing, little is known about how different coverage metrics could actually affect the fuzzing results in practice. More importantly, it is unclear whether there exists one coverage metric that is superior to all the other metrics. In this paper, we report the first systematic study on the impact of different coverage metrics in fuzzing. To this end, we formally define and discuss the concept of sensitivity, which can be used to theoretically compare different coverage metrics. We then present several coverage metrics with their variants. We conduct a study on these metrics with the DARPA CGC dataset, the LAVA-M dataset, and a set of real-world applications (a total of 221 binaries). We find that because each fuzzing instance has limited resources (time and computation power), (1) each metric has its unique merit in terms of flipping certain types of branches (thus vulnerability finding) and (2) there is no grand slam coverage metric that defeats all the others. We also explore combining different coverage metrics through cross-seeding, and the result is very encouraging: this pure fuzzing based approach can crash at least the same numbers of binaries in the CGC dataset as a previous approach (Driller) that combines fuzzing and concolic execution. At the same time, our approach uses fewer computing resources