Cross-modal food retrieval: Learning a joint embedding of food images and recipes with semantic consistency and attention mechanism

• Main Authors: WANG, Hao, SAHOO, Doyen, LIU, Chenghao, SHU, Ke, ACHANANUPARP, Palakorn, LIM, Ee-peng, HOI, Steven C. H.
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2022
• Subjects: Correlation; Cross-Modal Retrieval; Data models; Deep Learning; Semantics; Sugar; Task analysis; Training; Visionand-Language; Visualization; Artificial Intelligence and Robotics; Databases and Information Systems; Graphics and Human Computer Interfaces
• Online Access: https://ink.library.smu.edu.sg/sis_research/6268; https://ink.library.smu.edu.sg/context/sis_research/article/7271/viewcontent/cross_modal_food_retrieval.pdf
• Institution: Singapore Management University

Food retrieval is an important task to perform analysis of food-related information, where we are interested in retrieving relevant information about the queried food item such as ingredients, cooking instructions, etc. In this paper, we investigate cross-modal retrieval between food images and cooking recipes. The goal is to learn an embedding of images and recipes in a common feature space, such that the corresponding image-recipe embeddings lie close to one another. Two major challenges in addressing this problem are 1) large intra-variance and small inter-variance across cross-modal food data; and 2) difficulties in obtaining discriminative recipe representations. To address these two problems, we propose Semantic-Consistent and Attentionbased Networks (SCAN), which regularize the embeddings of the two modalities through aligning output semantic probabilities. Besides, we exploit a self-attention mechanism to improve the embedding of recipes.We evaluate the performance of the proposed method on the large-scale Recipe1M dataset, and show that we can outperform several state-of-the-art cross-modal retrieval strategies for food images and cooking recipes by a significant margin.