Data cleaning and refinement for code related AI systems

• Main Author: Tay, Arron Hong Yi
• Other Authors: Liu Yang
• Format: Final Year Project
• Language: English
• Published: Nanyang Technological University 2023
• Subjects: Engineering::Computer science and engineering::Computing methodologies::Artificial intelligence
• Online Access: https://hdl.handle.net/10356/166197
• Institution: Nanyang Technological University

This final-year project will cover data cleaning and refinement to improve the quality of data for various natural language processing (NLP) projects, such as code clone detection and code-to-text conversion. The project will focus on using the PyTorch library to train a masked language model to detect code clones in the POJ104 dataset and generate text from code for the code-text dataset. This will be achieved by using one of several pre-trained models, including OpenAI-GPT, BERT, RoBERTa, or DistilBERT. The refinement process will involve detailed data analysis and scripts to preprocess the data. Within these projects taken to do refinement, the project usually includes scripts which carry out loading, preprocessing code data, creating and training model, evaluation and lastly, testing against the answers. During the training phase, libraries such as transformers and tensorboard will be used to provide the pre-trained models and tokenizers, as well as visualizations for the training process model. The trained model will be evaluated at every epoch to check its performance of the training model. After the training phase, a model will be built to be tested against a set of answers to measure the performance model. Mean Average Precision and Smooth bleu4 Score will be used to measure performance for code-duplication project and code-to-text project, respectively. This report will cover the methods used to improve the data quality and discuss the techniques used for refinement which aims to improve the accuracy and efficiency of NLP projects for code that could benefit a wider range of applications in different industries.