Injecting descriptive meta-information into pre-trained language models with hypernetworks

• Main Authors: DUAN, Wenying, HE, Xiaoxi, ZHOU, Zimu, RAO, Hong, THIELE, Lothar
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2021
• Subjects: descriptive meta-information; hypernetworks; pre-trained language mode; Programming Languages and Compilers; Software Engineering
• Online Access: https://ink.library.smu.edu.sg/sis_research/6237; https://ink.library.smu.edu.sg/context/sis_research/article/7240/viewcontent/Injecting_Descriptive_Meta_Information_into_Pre_Trained_Language_Models_with_Hypernetworks.pdf
• Institution: Singapore Management University

Pre-trained language models have been widely adopted as backbones in various natural language processing tasks. However, existing pre-trained language models ignore the descriptive meta-information in the text such as the distinction between the title and the mainbody, leading to over-weighted attention to insignificant text. In this paper, we propose a hypernetwork-based architecture to model the descriptive meta-information and integrate it into pre-trained language models. Evaluations on three natural language processing tasks show that our method notably improves the performance of pre-trained language models and achieves the state-of-the-art results on keyphrase extraction.