Multimodal zero-shot hateful meme detection

• Main Authors: ZHU, Jiawen, LEE, Roy Ka-Wei, CHONG, Wen Haw
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2022
• Subjects: Hateful memes; Multimodal; Social media mining; Artificial Intelligence and Robotics; Databases and Information Systems; Graphics and Human Computer Interfaces
• Online Access: https://ink.library.smu.edu.sg/sis_research/8257; https://ink.library.smu.edu.sg/context/sis_research/article/9260/viewcontent/Multimodal_Zero_Shot_Hateful_Meme_Detection.pdf
• Institution: Singapore Management University

Facebook has recently launched the hateful meme detection challenge, which garnered much attention in academic and industry research communities. Researchers have proposed multimodal deep learning classification methods to perform hateful meme detection. While the proposed methods have yielded promising results, these classification methods are mostly supervised and heavily rely on labeled data that are not always available in the real-world setting. Therefore, this paper explores and aims to perform hateful meme detection in a zero-shot setting. Working towards this goal, we propose Target-Aware Multimodal Enhancement (TAME), which is a novel deep generative framework that can improve existing hateful meme classification models’ performance in detecting unseen types of hateful memes. We conduct extensive experiments on the Facebook hateful meme dataset, and the results show that TAME can significantly improve the state-of-the-art hateful meme classification methods’ performance in seen and unseen settings.