Multi-modal cooking workflow construction for food recipes

• Main Authors: PAN, Liangming, CHEN, Jingjing, WU, Jianlong, LIU, Shaoteng, NGO, Chong-wah, KAN, Min-Yen, JIANG, Yugang, CHUA, Tat-Seng
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2020
• Subjects: cause-and-effect reasoning; cooking workflow; deep learning; food recipes; mm-res dataset; multi-modal fusion; Databases and Information Systems; Graphics and Human Computer Interfaces
• Online Access: https://ink.library.smu.edu.sg/sis_research/6464; https://ink.library.smu.edu.sg/context/sis_research/article/7467/viewcontent/3394171.3413765.pdf
• Institution: Singapore Management University

Understanding food recipe requires anticipating the implicit causal effects of cooking actions, such that the recipe can be converted into a graph describing the temporal workflow of the recipe. This is a non-trivial task that involves common-sense reasoning. However, existing efforts rely on hand-crafted features to extract the workflow graph from recipes due to the lack of large-scale labeled datasets. Moreover, they fail to utilize the cooking images, which constitute an important part of food recipes. In this paper, we build MM-ReS, the first large-scale dataset for cooking workflow construction, consisting of 9,850 recipes with human-labeled workflow graphs. Cooking steps are multi-modal, featuring both text instructions and cooking images. We then propose a neural encoder–decoder model that utilizes both visual and textual information to construct the cooking workflow, which achieved over 20% performance gain over existing hand-crafted baselines.