Mixed-dish recognition with contextual relation networks

• Main Authors: DENG, Lixi, CHEN, Jingjing, SUN, Qianru, HE, Xiangnan, TANG, Sheng, MING, Zhaoyan, ZHANG, Yongdong, CHUA, Tat-Seng
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2019
• Subjects: Food image recognition; image contexts; object detection; Databases and Information Systems; Graphics and Human Computer Interfaces
• Online Access: https://ink.library.smu.edu.sg/sis_research/4448; https://ink.library.smu.edu.sg/context/sis_research/article/5451/viewcontent/mm19_mixed_dish.pdf
• Institution: Singapore Management University

Mixed dish is a food category that contains different dishes mixed in one plate, and is popular in Eastern and Southeast Asia. Recognizing individual dishes in a mixed dish image is important for health related applications, e.g. calculating the nutrition values. However, most existing methods that focus on single dish classification are not applicable to mixed-dish recognition. The new challenge in recognizing mixed-dish images are the complex ingredient combination and severe overlap among different dishes. In order to tackle these problems, we propose a novel approach called contextual relation networks (CR-Nets) that encodes the implicit and explicit contextual relations among multiple dishes using region-level features and label-level co-occurrence, respectively. This is inspired by the intuition that people are likely to choose dishes with common eating habits, e.g., with multiple nutrition but without repeating ingredients. In addition, we collect a large-scale dataset of mixed-dish images that contain 9,254 mixed-dish images from 6 school canteens in Singapore. Extensive experiments on both our dataset and a smaller-scale public dataset validate that our CR-Nets can achieve top performance for localizing the dishes and recognizing their food categories.