Diffusion models for natural language processing

The autoregressive method has long been fundamental in natural language processing (NLP), where models predict the next token based on previously generated context. While effective, this approach suffers from inherent limitations, including slow generation speed and sequential dependency, hindering...

وصف كامل

محفوظ في:
التفاصيل البيبلوغرافية
المؤلف الرئيسي: Hoang, Minh Nhat
مؤلفون آخرون: Luu Anh Tuan
التنسيق: Final Year Project
اللغة:English
منشور في: Nanyang Technological University 2024
الموضوعات:
الوصول للمادة أونلاين:https://hdl.handle.net/10356/174904
الوسوم: إضافة وسم
لا توجد وسوم, كن أول من يضع وسما على هذه التسجيلة!
المؤسسة: Nanyang Technological University
اللغة: English
الوصف
الملخص:The autoregressive method has long been fundamental in natural language processing (NLP), where models predict the next token based on previously generated context. While effective, this approach suffers from inherent limitations, including slow generation speed and sequential dependency, hindering its scalability and efficiency in large-scale applications. Non-autoregressive methods offer a promising solution by enabling parallel decoding and mitigating the sequential bottleneck. In recent years, diffusion models have proven its superior performance in continuous domain such as image and video, and have also recently emerged as a non-autoregressive alternative in NLP. This paper explores the application of diffusion models across various NLP tasks, ranging from language modeling to sequence generation. Through extensive experimentation, we demonstrate that diffusion models not only rival but surpass predominated autoregressive architecture such as the fine-tuned GPT-2 Large models. Moreover, our analysis reveals that diffusion models exhibit a remarkable ability to generate diverse outputs under identical settings, indicating their potential to revolutionize NLP tasks. In summary, this paper presents a comprehensive investigation into the application of diffusion models in NLP, shedding light on their efficacy and potential to address longstanding challenges associated with autoregressive methods. These findings underscore the significance of diffusion models as a viable alternative for enhancing efficiency and scalability in natural language processing tasks.