XAI4FL: enhancing spectrum-based fault localization with explainable artificial intelligence

• Main Authors: RATNADIRA WIDYASARI, PRANA, Gede Artha Azriadi, AGUS HARYONO, Stefanus, TIAN, Yuan, ZACHIARY, Hafil Noer, LO, David
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2022
• Subjects: Fault localization; Explainable artificial intelligence (XAI); Model-agnostic explanation technique; Spectrum-based fault localization; Testing and debugging; Software Engineering
• Online Access: https://ink.library.smu.edu.sg/sis_research/7639; https://ink.library.smu.edu.sg/context/sis_research/article/8642/viewcontent/929800a499.pdf
• Institution: Singapore Management University

Manually finding the program unit (e.g., class, method, or statement) responsible for a fault is tedious and time-consuming. To mitigate this problem, many fault localization techniques have been proposed. A popular family of such techniques is spectrum-based fault localization (SBFL), which takes program execution traces (spectra) of failed and passed test cases as input and applies a ranking formula to compute a suspiciousness score for each program unit. However, most existing SBFL techniques fail to consider two facts: 1) not all failed test cases contribute equally to a considered fault(s), and 2) program units collaboratively contribute to the failure/pass of each test case in different ways. In this study, we propose a novel idea that first models the SBFL task as a classification problem of predicting whether a test case will fail or pass based on spectra information on program units. We subsequently apply eXplainable Artificial Intelligence (XAI) techniques to infer the local importance of each program unit to the prediction of each executed test case. Applying XAI to the failed test case, we retrieve information about which program statements within the test case that are considered the most important (i.e., have the biggest effect in making the test case failed). Such a design can automatically learn the unique contributions of failed test cases to the suspiciousness of a program unit by learning the different and collaborative contributions of program units to each test case's executed result. As far as we know, this is the first XAI-supported SBFL approach. We evaluate the new approach on the Defects4J benchmark dataset. We compare the performance of our approach against five popular SBFL techniques: DStar, Tarantula, Barinel, Ochiai, and OP. We measure their performance using the Top-K and EXAM scores. In particular, we focus on the result of the Top-1, which importance has been highlighted in automated program repair domain, where the proposed methods often assume perfect fault localization (i.e., the fault must be found at the first rank of the suspiciousness list). Our results show that our approach, named XAI4FL, has a statistically significant and substantially better performance in terms of Top-1 than the SBFL approaches. We also compare our approach with a simpler approach to get feature importance in a tree-based model (i.e., using the Mean Decrease in Impurity method). Our results show that XAI4FL statistically significantly outperforms the MDI method in Top-K and EXAM score. Our results and findings highlight that the utilization of XAI for fault localization can improve the overall results of fault localization techniques.