Injecting descriptive meta-information into pre-trained language models with hypernetworks
Pre-trained language models have been widely adopted as backbones in various natural language processing tasks. However, existing pre-trained language models ignore the descriptive meta-information in the text such as the distinction between the title and the mainbody, leading to over-weighted atten...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | text |
| Language: | English |
| Published: |
Institutional Knowledge at Singapore Management University
2021
|
| Subjects: | |
| Online Access: | https://ink.library.smu.edu.sg/sis_research/6237 https://ink.library.smu.edu.sg/context/sis_research/article/7240/viewcontent/Injecting_Descriptive_Meta_Information_into_Pre_Trained_Language_Models_with_Hypernetworks.pdf |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Institution: | Singapore Management University |
| Language: | English |
| Summary: | Pre-trained language models have been widely adopted as backbones in various natural language processing tasks. However, existing pre-trained language models ignore the descriptive meta-information in the text such as the distinction between the title and the mainbody, leading to over-weighted attention to insignificant text. In this paper, we propose a hypernetwork-based architecture to model the descriptive meta-information and integrate it into pre-trained language models. Evaluations on three natural language processing tasks show that our method notably improves the performance of pre-trained language models and achieves the state-of-the-art results on keyphrase extraction. |
|---|