Abstractive summarization framework based on pre-training and contrastive learning

Abstractive summarization aims at generating sentences which can well cover the key information of the document. In this dissertation, we verify the effectiveness of a generation-evaluation model trained with contrastive learning, which generates a set of candidate summaries first and then evaluates...

Full description

Saved in:
Bibliographic Details
Main Author: Yi, Chenqi
Other Authors: Lihui Chen
Format: Thesis-Master by Coursework
Language:English
Published: Nanyang Technological University 2023
Subjects:
Online Access:https://hdl.handle.net/10356/165534
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Nanyang Technological University
Language: English
id sg-ntu-dr.10356-165534
record_format dspace
spelling sg-ntu-dr.10356-1655342023-07-04T16:17:13Z Abstractive summarization framework based on pre-training and contrastive learning Yi, Chenqi Lihui Chen School of Electrical and Electronic Engineering ELHCHEN@ntu.edu.sg Engineering::Electrical and electronic engineering Abstractive summarization aims at generating sentences which can well cover the key information of the document. In this dissertation, we verify the effectiveness of a generation-evaluation model trained with contrastive learning, which generates a set of candidate summaries first and then evaluates the candidates to select the best one. Conventional methods directly introduce pre-trained models by default as the backbone of summary evaluation model. However, what pre-training task is helpful for improving the performance of pre-trained models on downstream summary evaluation task is still an open question. We conduct a study on Inverse Cloze Task (ICT) to answer the question. For the backbone of evaluation model, we compare the results of different pre-trained models. We further adopt ICT as additional pre-training task to pre-train the model and utilize it as the backbone of the evaluation model. We also verify and analyze how the masking rate in ICT affects the downstream evaluation task. Experiments on XSum and CNN/Daily Mail show that the model with additional ICT pre-training outperforms other pre-training baselines. Master of Science (Signal Processing) 2023-03-29T01:00:07Z 2023-03-29T01:00:07Z 2023 Thesis-Master by Coursework Yi, C. (2023). Abstractive summarization framework based on pre-training and contrastive learning. Master's thesis, Nanyang Technological University, Singapore. https://hdl.handle.net/10356/165534 https://hdl.handle.net/10356/165534 en application/pdf Nanyang Technological University
institution Nanyang Technological University
building NTU Library
continent Asia
country Singapore
Singapore
content_provider NTU Library
collection DR-NTU
language English
topic Engineering::Electrical and electronic engineering
spellingShingle Engineering::Electrical and electronic engineering
Yi, Chenqi
Abstractive summarization framework based on pre-training and contrastive learning
description Abstractive summarization aims at generating sentences which can well cover the key information of the document. In this dissertation, we verify the effectiveness of a generation-evaluation model trained with contrastive learning, which generates a set of candidate summaries first and then evaluates the candidates to select the best one. Conventional methods directly introduce pre-trained models by default as the backbone of summary evaluation model. However, what pre-training task is helpful for improving the performance of pre-trained models on downstream summary evaluation task is still an open question. We conduct a study on Inverse Cloze Task (ICT) to answer the question. For the backbone of evaluation model, we compare the results of different pre-trained models. We further adopt ICT as additional pre-training task to pre-train the model and utilize it as the backbone of the evaluation model. We also verify and analyze how the masking rate in ICT affects the downstream evaluation task. Experiments on XSum and CNN/Daily Mail show that the model with additional ICT pre-training outperforms other pre-training baselines.
author2 Lihui Chen
author_facet Lihui Chen
Yi, Chenqi
format Thesis-Master by Coursework
author Yi, Chenqi
author_sort Yi, Chenqi
title Abstractive summarization framework based on pre-training and contrastive learning
title_short Abstractive summarization framework based on pre-training and contrastive learning
title_full Abstractive summarization framework based on pre-training and contrastive learning
title_fullStr Abstractive summarization framework based on pre-training and contrastive learning
title_full_unstemmed Abstractive summarization framework based on pre-training and contrastive learning
title_sort abstractive summarization framework based on pre-training and contrastive learning
publisher Nanyang Technological University
publishDate 2023
url https://hdl.handle.net/10356/165534
_version_ 1772825923055255552