Multimodal fashion knowledge extraction as captioning

• Main Authors: YUAN, Yifei, ZHANG, Wenxuan, DENG, Yang, LAM, Wai
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2023
• Subjects: Fashion industry; Fashion knowledge extraction; Knowledge extraction; Multi-modal; Multi-modal data; Multimodal data mining; Product images; Rich texts; Social media; Social media analysis; Databases and Information Systems
• Online Access: https://ink.library.smu.edu.sg/sis_research/9170; https://ink.library.smu.edu.sg/context/sis_research/article/10173/viewcontent/MultimodalFashionK_av.pdf
• Institution: Singapore Management University

Social media plays a significant role in boosting the fashion industry, where a massive amount of fashion-related posts are generated every day. In order to obtain the rich fashion information from the posts, we study the task of social media fashion knowledge extraction. Fashion knowledge, which typically consists of the occasion, person attributes, and fashion item information, can be effectively represented as a set of tuples. Most previous studies on fashion knowledge extraction are based on the fashion product images without considering the rich text information in social media posts. Existing work on fashion knowledge extraction in social media is classification-based and requires to manually determine a set of fashion knowledge categories in advance. In our work, we propose to cast the task as a captioning problem to capture the interplay of the multimodal post information. Specifically, we transform the fashion knowledge tuples into a natural language caption with a sentence transformation method. Our framework then aims to generate the sentence-based fashion knowledge directly from the social media post. Inspired by the big success of pre-trained models, we build our model based on a multimodal pre-trained generative model and design several auxiliary tasks for enhancing the knowledge extraction. Since there is no existing dataset which can be directly borrowed to our task, we introduce a dataset consisting of social media posts with manual fashion knowledge annotation. Extensive experiments are conducted to demonstrate the effectiveness of our model.