Demystifying faulty code: Step-by-step reasoning for explainable fault localization

Fault localization is a critical process that involves identifying specific program elements responsible for program failures. Manually pinpointing these elements, such as classes, methods, or statements, which are associated with a fault is laborious and time-consuming. To overcome this challenge,...

Full description

Saved in:
Bibliographic Details
Main Authors: WIDYASARI, Ratnadira, ANG, Jia Wei, NGUYEN, Truong Giang, SHARMA, Neil, LO, David
Format: text
Language:English
Published: Institutional Knowledge at Singapore Management University 2024
Subjects:
LLM
Online Access:https://ink.library.smu.edu.sg/sis_research/9257
https://ink.library.smu.edu.sg/context/sis_research/article/10257/viewcontent/2403.10507v1.pdf
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Singapore Management University
Language: English
id sg-smu-ink.sis_research-10257
record_format dspace
spelling sg-smu-ink.sis_research-102572024-09-02T06:36:22Z Demystifying faulty code: Step-by-step reasoning for explainable fault localization WIDYASARI, Ratnadira ANG, Jia Wei NGUYEN, Truong Giang SHARMA, Neil LO, David Fault localization is a critical process that involves identifying specific program elements responsible for program failures. Manually pinpointing these elements, such as classes, methods, or statements, which are associated with a fault is laborious and time-consuming. To overcome this challenge, various fault localization tools have been developed. These tools typically generate a ranked list of suspicious program elements. However, this information alone is insufficient. A prior study emphasized that automated fault localization should offer a rationale. In this study, we investigate the step-by-step reasoning for explainable fault localization. We explore the potential of Large Language Models (LLM) in assisting developers in reasoning about code. We proposed FuseFL that utilizes several combinations of information to enhance the LLM results which are spectrum-based fault localization results, test case execution outcomes, and code description (i.e., explanation of what the given code is intended to do). We conducted our investigation using faulty code from Refactory dataset. First, we evaluate the performance of the automated fault localization. Our results demonstrate a 32.3 % increase in the number of successfully localized faults at Top-1 compared to the baseline. To evaluate the explanations generated by FuseFL, we create a dataset of human explanations that provide step-by-step reasoning as to why specific lines of code are considered faulty. This dataset consists of 324 faulty code files, along with explanations for 600 faulty lines. Furthermore, we also conducted human studies to evaluate the explanations. We found that for 22 out of the 30 randomly sampled cases, FuseFL generated correct explanations. 2024-03-01T08:00:00Z text application/pdf https://ink.library.smu.edu.sg/sis_research/9257 info:doi/10.1109/SANER60148.2024.00064 https://ink.library.smu.edu.sg/context/sis_research/article/10257/viewcontent/2403.10507v1.pdf http://creativecommons.org/licenses/by-nc-nd/4.0/ Research Collection School Of Computing and Information Systems eng Institutional Knowledge at Singapore Management University fault localization explanation dataset LLM ChatGPT Software Engineering
institution Singapore Management University
building SMU Libraries
continent Asia
country Singapore
Singapore
content_provider SMU Libraries
collection InK@SMU
language English
topic fault localization
explanation
dataset
LLM
ChatGPT
Software Engineering
spellingShingle fault localization
explanation
dataset
LLM
ChatGPT
Software Engineering
WIDYASARI, Ratnadira
ANG, Jia Wei
NGUYEN, Truong Giang
SHARMA, Neil
LO, David
Demystifying faulty code: Step-by-step reasoning for explainable fault localization
description Fault localization is a critical process that involves identifying specific program elements responsible for program failures. Manually pinpointing these elements, such as classes, methods, or statements, which are associated with a fault is laborious and time-consuming. To overcome this challenge, various fault localization tools have been developed. These tools typically generate a ranked list of suspicious program elements. However, this information alone is insufficient. A prior study emphasized that automated fault localization should offer a rationale. In this study, we investigate the step-by-step reasoning for explainable fault localization. We explore the potential of Large Language Models (LLM) in assisting developers in reasoning about code. We proposed FuseFL that utilizes several combinations of information to enhance the LLM results which are spectrum-based fault localization results, test case execution outcomes, and code description (i.e., explanation of what the given code is intended to do). We conducted our investigation using faulty code from Refactory dataset. First, we evaluate the performance of the automated fault localization. Our results demonstrate a 32.3 % increase in the number of successfully localized faults at Top-1 compared to the baseline. To evaluate the explanations generated by FuseFL, we create a dataset of human explanations that provide step-by-step reasoning as to why specific lines of code are considered faulty. This dataset consists of 324 faulty code files, along with explanations for 600 faulty lines. Furthermore, we also conducted human studies to evaluate the explanations. We found that for 22 out of the 30 randomly sampled cases, FuseFL generated correct explanations.
format text
author WIDYASARI, Ratnadira
ANG, Jia Wei
NGUYEN, Truong Giang
SHARMA, Neil
LO, David
author_facet WIDYASARI, Ratnadira
ANG, Jia Wei
NGUYEN, Truong Giang
SHARMA, Neil
LO, David
author_sort WIDYASARI, Ratnadira
title Demystifying faulty code: Step-by-step reasoning for explainable fault localization
title_short Demystifying faulty code: Step-by-step reasoning for explainable fault localization
title_full Demystifying faulty code: Step-by-step reasoning for explainable fault localization
title_fullStr Demystifying faulty code: Step-by-step reasoning for explainable fault localization
title_full_unstemmed Demystifying faulty code: Step-by-step reasoning for explainable fault localization
title_sort demystifying faulty code: step-by-step reasoning for explainable fault localization
publisher Institutional Knowledge at Singapore Management University
publishDate 2024
url https://ink.library.smu.edu.sg/sis_research/9257
https://ink.library.smu.edu.sg/context/sis_research/article/10257/viewcontent/2403.10507v1.pdf
_version_ 1814047846798721024