Towards generalizable neural program understanding and synthesis

• Main Author: Li, Zhiming
• Other Authors: Liu Yang
• Format: Thesis-Doctor of Philosophy
• Language: English
• Published: Nanyang Technological University 2024
• Subjects: Computer and Information Science
• Online Access: https://hdl.handle.net/10356/181413
• Institution: Nanyang Technological University

The recent rapid advancement of deep learning, especially the development of the large language models (LLMs) has revolutionized the community of software engineering. Thanks to the great amount of open-sourced code-related data readily available for training, the neural network solutions manage to achieve great performance without considerable manual effort compared to the traditional rule-based tools. Similar to the natural language processing (NLP) community, the code-related tasks can be majorly categorized into (1) program understanding tasks (i.e., classification tasks such as vulnerability detection and type inference) and (2) program synthesis tasks, which aims to synthesize programs based on given specifications. Despite the great success achieved on neural program understanding and synthesis, the current state-of-the-art models are far from perfect and there are still many open questions remained. One of the most critical problems is the poor generalization of the current state-of-the-art models. In this thesis, we aim to develop more generalizable neural program understanding systems and neural program synthesis systems. First, we aim to investigate why the state-of-the-art neural program understanding models generalize poorly under the real-world inter-project setting. We focus on two popular classification tasks, namely, vulnerability detection and type inference. To achieve this purpose, we conduct a comprehensive bias behavior analysis for the encoder-only LLMs, which is the first of its kind. We further propose a novel measurement for the interpretation of the models' behavior. Besides, a useful bias mitigation mechanism (BPR) is introduced. The idea is to explicitly regularize the model behavior during training by identifying common logic properties among samples. Experimental results validate that BPR can effectively improve the generalization and robustness of the LLMs while not sacrificing the IID performance. Second, inspired by the lack of generalization of the previous imperative program synthesis paradigm for reinforcement learning. We propose GALOIS, a novel generalized logic synthesis framework based on a new sketch-based hybrid programming language that combines language constructs from both imperative programming and logic programming. Extensive evaluations on various complex tasks demonstrate the superiority of GALOIS over mainstream deep reinforcement learning and program synthesis baselines regarding the learning ability, generalizability, interpretability, and knowledge (logic) reusability across tasks. Third, despite the improvement in terms of generalization logic program synthesis can bring about, the current state-of-the-art logic program induction models are still far from perfect. Therefore, we aim to further improve the generalization and performance of the state-of-the-art neural program induction models. Motivated by the inefficient learning process of the neural logic machines, during which the models tend to repeat similar mistakes. We propose a novel regularization framework called FRGR, which can timely detect models’ repetition of similar errors, identify the root cause subprogram, and use it to regularize the model. FRGR manages to considerably improve the performance, generalization, training efficiency, and data efficiency of the NLMs. Finally, in spite of the improvement regarding generalization logic program synthesis introduced, the white-box program synthesizers are still unscalable and hard to generalize. One critical reason is that the domain-specific language (DSL), on which the program synthesis is conducted is hard to design and requires significant human expertise. For many real-world tasks, whose data is fuzzy and noisy, such as quantitative trading, a manual complete design of DSL becomes almost impossible. In the meanwhile, a pure black-box deep neural network strategy can automatically discover useful patterns without significant manual effort, whereas it is poor in terms of generalization and interpretability. In order to make the best of both worlds, we propose a novel neural-symbolic framework called program sketch-based tuning (PST) for the quantitative trading tasks. The results demonstrate that PST can significantly improve the generalization of the DRL trading strategies while being extremely lightweight.