Cross-thought for sentence encoder pre-training

• Main Authors: WANG, Shuohang, FANG, Yuwei, SUN, Siqi, GAN, Zhe, CHENG, Yu, LIU, Jingjing, JIANG, Jing
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2020
• Subjects: Databases and Information Systems; Programming Languages and Compilers
• Online Access: https://ink.library.smu.edu.sg/sis_research/5602; https://ink.library.smu.edu.sg/context/sis_research/article/6605/viewcontent/EMNLP_2020a.pdf
• Institution: Singapore Management University

In this paper, we propose Cross-Thought, a novel approach to pre-training sequence encoder, which is instrumental in building reusable sequence embeddings for large-scale NLP tasks such as question answering. Instead of using the original signals of full sentences, we train a Transformer-based seque...