MERMAID: A dataset and framework for multimodal meme semantic understanding

• Main Authors: TOH, Shaun, KUEK, Adriel, CHONG, Wen Haw, LEE, Roy Ka Wei
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2023
• Subjects: Memes; Multimodal; Semantic Extraction; Databases and Information Systems; Graphics and Human Computer Interfaces; Mental and Social Health
• Online Access: https://ink.library.smu.edu.sg/sis_research/8746; https://ink.library.smu.edu.sg/context/sis_research/article/9749/viewcontent/MERMAID_av.pdf
• Institution: Singapore Management University

Memes are widely used to convey cultural and societal issues and have a significant impact on public opinion. However, little work has been done on understanding and explaining the semantics expressed in multimodal memes. To fill this research gap, we introduce MERMAID, a dataset consisting of 3,633 memes annotated with their entities and relations, and propose a novel MERF pipeline that extracts entities and their relationships in memes. Our framework combines state-of-the-art techniques from natural language processing and computer vision to extract text and image features and infer relationships between entities in memes. We evaluate the proposed framework on a real-world meme dataset and establish the benchmark for the new multimodal meme semantic understanding task. Our evaluation also includes a low-resource setting, where we assess the applicability of our framework to low-resource settings, which is a common problem due to the high cost and lack of labeled data for relations in memes. Overall, our work contributes to the understanding of the semantics of memes, a crucial form of communication in today's society.