Two sides of the same coin: Exploiting the impact of identifiers in neural code comprehension

Previous studies have demonstrated that neural code comprehension models are vulnerable to identifier naming. By renaming as few as one identifier in the source code, the models would output completely irrelevant results, indicating that identifiers can be misleading for model prediction. However, i...

Full description

Saved in:
Bibliographic Details
Main Authors: GAO, Shuzheng, GAO, Cuiyun, WANG, Chaozheng, SUN, Jun, LO, David, YU, Yue
Format: text
Language:English
Published: Institutional Knowledge at Singapore Management University 2023
Subjects:
Online Access:https://ink.library.smu.edu.sg/sis_research/9270
https://ink.library.smu.edu.sg/context/sis_research/article/10270/viewcontent/TwoSidesCoin_2023_av.pdf
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Singapore Management University
Language: English
id sg-smu-ink.sis_research-10270
record_format dspace
spelling sg-smu-ink.sis_research-102702024-09-09T07:02:36Z Two sides of the same coin: Exploiting the impact of identifiers in neural code comprehension GAO, Shuzheng GAO, Cuiyun WANG, Chaozheng SUN, Jun LO, David YU, Yue Previous studies have demonstrated that neural code comprehension models are vulnerable to identifier naming. By renaming as few as one identifier in the source code, the models would output completely irrelevant results, indicating that identifiers can be misleading for model prediction. However, identifiers are not completely detrimental to code comprehension, since the semantics of identifier names can be related to the program semantics. Well exploiting the two opposite impacts of identifiers is essential for enhancing the robustness and accuracy of neural code comprehension, and still remains under-explored. In this work, we propose to model the impact of identifiers from a novel causal perspective, and propose a counterfactual reasoning-based framework named CREAM. CREAM explicitly captures the misleading information of identifiers through multi-task learning in the training stage, and reduces the misleading impact by counterfactual inference in the inference stage. We evaluate CREAM on three popular neural code comprehension tasks, including function naming, defect detection and code classification. Experiment results show that CREAM not only significantly outperforms baselines in terms of robustness (e.g., +37.9% on the function naming task at F1 score), but also achieve improved results on the original datasets (e.g., +0.5% on the function naming task at F1 score). 2023-05-01T07:00:00Z text application/pdf https://ink.library.smu.edu.sg/sis_research/9270 info:doi/10.1109/ICSE48619.2023.00164 https://ink.library.smu.edu.sg/context/sis_research/article/10270/viewcontent/TwoSidesCoin_2023_av.pdf http://creativecommons.org/licenses/by-nc-nd/4.0/ Research Collection School Of Computing and Information Systems eng Institutional Knowledge at Singapore Management University Code comprehension Comprehension models Counterfactuals F1 scores Misleading informations Model prediction Multitask learning Neural code Program semantics Source codes Software Engineering
institution Singapore Management University
building SMU Libraries
continent Asia
country Singapore
Singapore
content_provider SMU Libraries
collection InK@SMU
language English
topic Code comprehension
Comprehension models
Counterfactuals
F1 scores
Misleading informations
Model prediction
Multitask learning
Neural code
Program semantics
Source codes
Software Engineering
spellingShingle Code comprehension
Comprehension models
Counterfactuals
F1 scores
Misleading informations
Model prediction
Multitask learning
Neural code
Program semantics
Source codes
Software Engineering
GAO, Shuzheng
GAO, Cuiyun
WANG, Chaozheng
SUN, Jun
LO, David
YU, Yue
Two sides of the same coin: Exploiting the impact of identifiers in neural code comprehension
description Previous studies have demonstrated that neural code comprehension models are vulnerable to identifier naming. By renaming as few as one identifier in the source code, the models would output completely irrelevant results, indicating that identifiers can be misleading for model prediction. However, identifiers are not completely detrimental to code comprehension, since the semantics of identifier names can be related to the program semantics. Well exploiting the two opposite impacts of identifiers is essential for enhancing the robustness and accuracy of neural code comprehension, and still remains under-explored. In this work, we propose to model the impact of identifiers from a novel causal perspective, and propose a counterfactual reasoning-based framework named CREAM. CREAM explicitly captures the misleading information of identifiers through multi-task learning in the training stage, and reduces the misleading impact by counterfactual inference in the inference stage. We evaluate CREAM on three popular neural code comprehension tasks, including function naming, defect detection and code classification. Experiment results show that CREAM not only significantly outperforms baselines in terms of robustness (e.g., +37.9% on the function naming task at F1 score), but also achieve improved results on the original datasets (e.g., +0.5% on the function naming task at F1 score).
format text
author GAO, Shuzheng
GAO, Cuiyun
WANG, Chaozheng
SUN, Jun
LO, David
YU, Yue
author_facet GAO, Shuzheng
GAO, Cuiyun
WANG, Chaozheng
SUN, Jun
LO, David
YU, Yue
author_sort GAO, Shuzheng
title Two sides of the same coin: Exploiting the impact of identifiers in neural code comprehension
title_short Two sides of the same coin: Exploiting the impact of identifiers in neural code comprehension
title_full Two sides of the same coin: Exploiting the impact of identifiers in neural code comprehension
title_fullStr Two sides of the same coin: Exploiting the impact of identifiers in neural code comprehension
title_full_unstemmed Two sides of the same coin: Exploiting the impact of identifiers in neural code comprehension
title_sort two sides of the same coin: exploiting the impact of identifiers in neural code comprehension
publisher Institutional Knowledge at Singapore Management University
publishDate 2023
url https://ink.library.smu.edu.sg/sis_research/9270
https://ink.library.smu.edu.sg/context/sis_research/article/10270/viewcontent/TwoSidesCoin_2023_av.pdf
_version_ 1814047869962813440