Towards generalizable neural program understanding and synthesis - Li, Zhiming (Nanyang Technological University, 2024)
The recent rapid advancement of deep learning, especially the development of the large language models (LLMs) has revolutionized the community of software engineering. Thanks to the great amount of open-sourced code-related data readily available for training, the neural network solutions manage to achieve great performance without considerable manual effort compared to the traditional rule-based tools. Similar to the natural language processing (NLP) community, the code-related tasks can be majorly categorized into (1) program understanding tasks (i.e., classification tasks such as vulnerability detection and type inference) and (2) program synthesis tasks, which aims to synthesize programs based on given specifications. Despite the great success achieved on neural program understanding and synthesis, the current state-of-the-art models are far from perfect and there are still many open questions remained. One of the most critical problems is the poor generalization of the current state-of-the-art models. In this thesis, we aim to develop more generalizable neural program understanding systems and neural program synthesis systems. First, we aim to investigate why the state-of-the-art neural program understanding models generalize poorly under the real-world inter-project setting. We focus on two popular classification tasks, namely, vulnerability detection and type inference. To achieve this purpose, we conduct a comprehensive bias behavior analysis for the encoder-only LLMs, which is the first of its kind. We further propose a novel measurement for the interpretation of the models' behavior. Besides, a useful bias mitigation mechanism (BPR) is introduced. The idea is to explicitly regularize the model behavior during training by identifying common logic properties among samples. Experimental results validate that BPR can effectively improve the generalization and robustness of the LLMs while not sacrificing the IID performance.