Simple and effective curriculum pointer-generator networks for reading comprehension over long narratives

• Main Authors: TAY, Yi, WANG, Shuohang, LUU, Anh Tuan, FU, Jie, PHAN, Minh C., YUAN, Xingdi, RAO, Jinfeng, HUI, Siu Cheung, ZHANG, Aston
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2019
• Subjects: OS and Networks
• Online Access: https://ink.library.smu.edu.sg/scis_studentpub/1; https://ink.library.smu.edu.sg/context/scis_studentpub/article/1000/viewcontent/P19_1486.pdf
• Institution: Singapore Management University

This paper tackles the problem of reading comprehension over long narratives where documents easily span over thousands of tokens. We propose a curriculum learning (CL) based Pointer-Generator framework for reading/sampling over large documents, enabling diverse training of the neural model based on the notion of alternating contextual difficulty. This can be interpreted as a form of domain randomization and/or generative pretraining during training. To this end, the usage of the Pointer-Generator softens the requirement of having the answer within the context, enabling us to construct diverse training samples for learning. Additionally, we propose a new Introspective Alignment Layer (IAL), which reasons over decomposed alignments using block-based self-attention. We evaluate our proposed method on the NarrativeQA reading comprehension benchmark, achieving state-of-the-art performance, improving existing baselines by 51% relative improvement on BLEU-4 and 17% relative improvement on Rouge-L. Extensive ablations confirm the effectiveness of our proposed IAL and CL components.