Improving multimodal named entity recognition via entity span detection with unified multimodal transformer

• Main Authors: YU, Jianfei, Jing JIANG, YANG, Li, XIA, Rui
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2020
• Subjects: Databases and Information Systems; Theory and Algorithms
• Online Access: https://ink.library.smu.edu.sg/sis_research/5272; https://ink.library.smu.edu.sg/context/sis_research/article/6275/viewcontent/8._Improving_Multimodal_Named_Entity_Recognition_via_Entity_Span_Detection_with_United_Multimodal_Transformer__ACL2020_.pdf
• Institution: Singapore Management University

In this paper, we study Multimodal Named Entity Recognition (MNER) for social media posts. Existing approaches for MNER mainly suffer from two drawbacks: (1) despite generating word-aware visual representations, their word representations are insensitive to the visual context; (2) most of them ignore the bias brought by the visual context. To tackle the first issue, we propose a multimodal interaction module to obtain both image-aware word representations and word-aware visual representations. To alleviate the visual bias, we further propose to leverage purely text-based entity span detection as an auxiliary module, and design a Unified Multimodal Transformer to guide the final predictions with the entity span predictions. Experiments show that our unified approach achieves the new state-of-the-art performance on two benchmark datasets.