Pro-Cap: Leveraging a frozen vision-language model for hateful meme detection

• Main Authors: CAO, Rui, HEE, Ming Shan, KUEK, Adriel, CHONG, Wen Haw, LEE, Roy Ka-Wei, JIANG, Jing
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2023
• Subjects: Memes; multimodal; semantic extraction; Databases and Information Systems; Graphic Communications; Graphics and Human Computer Interfaces
• Online Access: https://ink.library.smu.edu.sg/sis_research/8477; https://ink.library.smu.edu.sg/context/sis_research/article/9480/viewcontent/Pro_Cap_pvoa_cc_by.pdf
• Institution: Singapore Management University

Hateful meme detection is a challenging multimodal task that requires comprehension of both vision and language, as well as cross-modal interactions. Recent studies have tried to fine-tune pre-trained vision-language models (PVLMs) for this task. However, with increasing model sizes, it becomes impo...