Learning from web recipe-image pairs for food recognition: Problem, baselines and performance

• Main Authors: ZHU, Bin, NGO, Chong-wah, CHAN, Wing-Kwong
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2022
• Subjects: Image recognition;Training;Generative adversarial networks;Feature extraction;Visualization;Data models;Context modeling;Food recognition;image-to-recipe retrieval;image-to-image retrieval; Databases and Information Systems
• Online Access: https://ink.library.smu.edu.sg/sis_research/7246; https://ink.library.smu.edu.sg/context/sis_research/article/8249/viewcontent/tmm2021_zhu_ngo_chan.pdf
• Institution: Singapore Management University

Cross-modal recipe retrieval has recently been explored for food recognition and understanding. Text-rich recipe provides not only visual content information (e.g., ingredients, dish presentation) but also procedure of food preparation (cutting and cooking styles). The paired data is leveraged to train deep models to retrieve recipes for food images. Most recipes on the Web include sample pictures as the references. The paired multimedia data is not noise-free, due to errors such as pairing of images containing partially prepared dishes with recipes. The content of recipes and food images are not always consistent due to free-style writing and preparation of food in different environments. As a consequence, the effectiveness of learning cross-modal deep models from such noisy web data is questionable. This paper conducts an empirical study to provide insights whether the features learnt with noisy pair data are resilient and could capture the modality correspondence between visual and text.