Diffusion models for natural language processing

• Main Author: Hoang, Minh Nhat
• Other Authors: Luu Anh Tuan
• Format: Final Year Project
• Language: English
• Published: Nanyang Technological University 2024
• Subjects: Computer and Information Science; Natural language processing
• Online Access: https://hdl.handle.net/10356/174904
• Institution: Nanyang Technological University

The autoregressive method has long been fundamental in natural language processing (NLP), where models predict the next token based on previously generated context. While effective, this approach suffers from inherent limitations, including slow generation speed and sequential dependency, hindering its scalability and efficiency in large-scale applications. Non-autoregressive methods offer a promising solution by enabling parallel decoding and mitigating the sequential bottleneck. In recent years, diffusion models have proven its superior performance in continuous domain such as image and video, and have also recently emerged as a non-autoregressive alternative in NLP. This paper explores the application of diffusion models across various NLP tasks, ranging from language modeling to sequence generation. Through extensive experimentation, we demonstrate that diffusion models not only rival but surpass predominated autoregressive architecture such as the fine-tuned GPT-2 Large models. Moreover, our analysis reveals that diffusion models exhibit a remarkable ability to generate diverse outputs under identical settings, indicating their potential to revolutionize NLP tasks. In summary, this paper presents a comprehensive investigation into the application of diffusion models in NLP, shedding light on their efficacy and potential to address longstanding challenges associated with autoregressive methods. These findings underscore the significance of diffusion models as a viable alternative for enhancing efficiency and scalability in natural language processing tasks.