A hybrid approach for detecting prerequisite relations in multi-modal food recipes

• Main Authors: PAN, Liangming, CHEN, Jingjing, LIU, Shaoteng, NGO, Chong-wah, KAN, Min-Yen, CHUA, Tat-Seng
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2020
• Subjects: Feature extraction; Training; Task analysis; Semantics; Pipelines; Deep learning; Predictive models; Food recipes; cooking workflow; prerequisite trees; multi-modal fusion; cause-and-effect reasoning; deep learning; Databases and Information Systems
• Online Access: https://ink.library.smu.edu.sg/sis_research/7927; https://ink.library.smu.edu.sg/context/sis_research/article/8930/viewcontent/TMM20_Paper__1_.pdf
• Institution: Singapore Management University

Modeling the structure of culinary recipes is the core of recipe representation learning. Current approaches mostly focus on extracting the workflow graph from recipes based on text descriptions. Process images, which constitute an important part of cooking recipes, has rarely been investigated in recipe structure modeling. We study this recipe structure problem from a multi-modal learning perspective, by proposing a prerequisite tree to represent recipes with cooking images at a step-level granularity. We propose a simple-yet-effective two-stage framework to automatically construct the prerequisite tree for a recipe by (1) utilizing a trained classifier to detect pairwise prerequisite relations that fuses multi-modal features as input; then (2) applying different strategies (greedy method, maximum weight, and beam search) to build the tree structure. Experiments on the MM-ReS dataset demonstrates the advantages of introducing process images for recipe structure modeling. Also, compared with neural methods which require large numbers of training data, we show that our two-stage pipeline can achieve promising results using only 400 labeled prerequisite trees as training data.