Exploring the potential of ChatGPT in automated code refinement: An empirical study
Code review is an essential activity for ensuring the quality and maintainability of software projects. However, it is a time-consuming and often error-prone task that can significantly impact the development process. Recently, ChatGPT, a cutting-edge language model, has demonstrated impressive perf...
Saved in:
Main Authors: | , , , , , , |
---|---|
Format: | text |
Language: | English |
Published: |
Institutional Knowledge at Singapore Management University
2024
|
Subjects: | |
Online Access: | https://ink.library.smu.edu.sg/sis_research/8734 https://ink.library.smu.edu.sg/context/sis_research/article/9737/viewcontent/Potential_ChatGPT_CodeRefine_pv.pdf |
Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
Institution: | Singapore Management University |
Language: | English |
id |
sg-smu-ink.sis_research-9737 |
---|---|
record_format |
dspace |
spelling |
sg-smu-ink.sis_research-97372024-04-18T07:26:32Z Exploring the potential of ChatGPT in automated code refinement: An empirical study GUO, Qi LIU, Shangqing CAO, Junming LI, Xiaohong PENG, Xin XIE, Xiaofei CHEN, Bihuan Code review is an essential activity for ensuring the quality and maintainability of software projects. However, it is a time-consuming and often error-prone task that can significantly impact the development process. Recently, ChatGPT, a cutting-edge language model, has demonstrated impressive performance in various natural language processing tasks, suggesting its potential to automate code review processes. However, it is still unclear how well ChatGPT performs in code review tasks. To fill this gap, in this paper, we conduct the first empirical study to understand the capabilities of ChatGPT in code review tasks, specifically focusing on automated code refinement based on given code reviews. To conduct the study, we select the existing benchmark CodeReview and construct a new code review dataset with high quality. We use CodeReviewer, a state-of-the-art code review tool, as a baseline for comparison with ChatGPT. Our results show that ChatGPT outperforms CodeReviewer in code refinement tasks. Specifically, our results show that ChatGPT achieves higher EM and BLEU scores of 22.78 and 76.44 respectively, while the state-of-the-art method achieves only 15.50 and 62.88 on a high-quality code review dataset. We further identify the root causes for ChatGPT’s underperformance and propose several strategies to mitigate these challenges. Our study provides insights into the potential of ChatGPT in automating the code review process, and highlights the potential research directions. 2024-04-01T07:00:00Z text application/pdf https://ink.library.smu.edu.sg/sis_research/8734 info:doi/10.1145/3597503.3623306 https://ink.library.smu.edu.sg/context/sis_research/article/9737/viewcontent/Potential_ChatGPT_CodeRefine_pv.pdf http://creativecommons.org/licenses/by-nc-nd/4.0/ Research Collection School Of Computing and Information Systems eng Institutional Knowledge at Singapore Management University Software development techniques Software maintenance tools ChatGPT Artificial Intelligence and Robotics Software Engineering |
institution |
Singapore Management University |
building |
SMU Libraries |
continent |
Asia |
country |
Singapore Singapore |
content_provider |
SMU Libraries |
collection |
InK@SMU |
language |
English |
topic |
Software development techniques Software maintenance tools ChatGPT Artificial Intelligence and Robotics Software Engineering |
spellingShingle |
Software development techniques Software maintenance tools ChatGPT Artificial Intelligence and Robotics Software Engineering GUO, Qi LIU, Shangqing CAO, Junming LI, Xiaohong PENG, Xin XIE, Xiaofei CHEN, Bihuan Exploring the potential of ChatGPT in automated code refinement: An empirical study |
description |
Code review is an essential activity for ensuring the quality and maintainability of software projects. However, it is a time-consuming and often error-prone task that can significantly impact the development process. Recently, ChatGPT, a cutting-edge language model, has demonstrated impressive performance in various natural language processing tasks, suggesting its potential to automate code review processes. However, it is still unclear how well ChatGPT performs in code review tasks. To fill this gap, in this paper, we conduct the first empirical study to understand the capabilities of ChatGPT in code review tasks, specifically focusing on automated code refinement based on given code reviews. To conduct the study, we select the existing benchmark CodeReview and construct a new code review dataset with high quality. We use CodeReviewer, a state-of-the-art code review tool, as a baseline for comparison with ChatGPT. Our results show that ChatGPT outperforms CodeReviewer in code refinement tasks. Specifically, our results show that ChatGPT achieves higher EM and BLEU scores of 22.78 and 76.44 respectively, while the state-of-the-art method achieves only 15.50 and 62.88 on a high-quality code review dataset. We further identify the root causes for ChatGPT’s underperformance and propose several strategies to mitigate these challenges. Our study provides insights into the potential of ChatGPT in automating the code review process, and highlights the potential research directions. |
format |
text |
author |
GUO, Qi LIU, Shangqing CAO, Junming LI, Xiaohong PENG, Xin XIE, Xiaofei CHEN, Bihuan |
author_facet |
GUO, Qi LIU, Shangqing CAO, Junming LI, Xiaohong PENG, Xin XIE, Xiaofei CHEN, Bihuan |
author_sort |
GUO, Qi |
title |
Exploring the potential of ChatGPT in automated code refinement: An empirical study |
title_short |
Exploring the potential of ChatGPT in automated code refinement: An empirical study |
title_full |
Exploring the potential of ChatGPT in automated code refinement: An empirical study |
title_fullStr |
Exploring the potential of ChatGPT in automated code refinement: An empirical study |
title_full_unstemmed |
Exploring the potential of ChatGPT in automated code refinement: An empirical study |
title_sort |
exploring the potential of chatgpt in automated code refinement: an empirical study |
publisher |
Institutional Knowledge at Singapore Management University |
publishDate |
2024 |
url |
https://ink.library.smu.edu.sg/sis_research/8734 https://ink.library.smu.edu.sg/context/sis_research/article/9737/viewcontent/Potential_ChatGPT_CodeRefine_pv.pdf |
_version_ |
1814047497189851136 |