APPLICATION OF BERTOPIC ON BERTSUM MODEL FOR INDONESIAN ABSTRACTIVE TEXT SUMMARIZATION - Satya Putra Mahendra, Farrel
Text summarization is a field in natural language processing that aims to generate summaries by removing redundant information, creating a shorter version while retaining essential content. Text summarization can be divided into two types: extractive and abstractive. This research focuses on abstractive summarization, which generates summaries by rephrasing the main information in a style different from the original text. In recent years, neural-based models, particularly those using transformer architectures, have gained popularity in text summarization. One of the leading models is BERTSum, which uses BERT as an encoder to obtain feature representations from the document. BERTSum has proven to outperform other abstractive text summarization models. Evaluating abstractive text summarization requires a specially designed dataset with a high proportion of novel n-grams in the summary labels, indicating that the summary content is unique and original. Therefore, this study uses the XL-Sum Indonesia dataset, which has the highest percentage of novel n-grams among other Indonesian datasets, as a basis for developing an abstractive text summarization model. Research on BERT for Indonesian text summarization remains limited. Previous studies comparing Indonesian BERT and English BERT concluded that the performance of English BERT is superior to that of Indonesian BERT. Theoretically, the performance of Indonesian BERT should be better because it undergoes pretraining using an Indonesian corpus. This study aims to investigate the reasons behind this phenomenon. Transformer-based models like BERTSum have advantages, but they also have limitations in the number of tokens that can be used as input. BERT has a limit of 512 tokens, which often results in document truncation and the loss of important information, especially in long documents. Topic modeling, which identifies hidden topics in documents, can help address this issue by capturing the document's global semantics. The combination of topic models and transformers can improve the model's understanding of the entire document.