Decomposing generation networks with structure prediction for recipe generation
Recipe generation from food images and ingredients is a challenging task, which requires the interpretation of the information from another modality. Different from the image captioning task, where the captions usually have one sentence, cooking instructions contain multiple sentences and have obvio...
Saved in:
Main Authors: | , , , |
---|---|
Format: | text |
Language: | English |
Published: |
Institutional Knowledge at Singapore Management University
2022
|
Subjects: | |
Online Access: | https://ink.library.smu.edu.sg/sis_research/6962 https://ink.library.smu.edu.sg/context/sis_research/article/7965/viewcontent/DecomposingCHRecipe_av.pdf |
Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
Institution: | Singapore Management University |
Language: | English |
id |
sg-smu-ink.sis_research-7965 |
---|---|
record_format |
dspace |
spelling |
sg-smu-ink.sis_research-79652022-03-04T05:56:58Z Decomposing generation networks with structure prediction for recipe generation WANG, Hao LIN, Guosheng HOI, Steven C. H. MIAO, Chunyan Recipe generation from food images and ingredients is a challenging task, which requires the interpretation of the information from another modality. Different from the image captioning task, where the captions usually have one sentence, cooking instructions contain multiple sentences and have obvious structures. To help the model capture the recipe structure and avoid missing some cooking details, we propose a novel framework: Decomposing Generation Networks (DGN) with structure prediction, to get more structured and complete recipe generation outputs. Specifically, we split each cooking instruction into several phases, and assign different sub-generators to each phase. Our approach includes two novel ideas: (i) learning the recipe structures with the global structure prediction component and (ii) producing recipe phases in the sub-generator output component based on the predicted structure. Extensive experiments on the challenging large-scale Recipe1M dataset validate the effectiveness of our proposed model, which improves the performance over the state-of-the-art results. 2022-06-01T07:00:00Z text application/pdf https://ink.library.smu.edu.sg/sis_research/6962 info:doi/10.1016/j.patcog.2022.108578 https://ink.library.smu.edu.sg/context/sis_research/article/7965/viewcontent/DecomposingCHRecipe_av.pdf http://creativecommons.org/licenses/by-nc-nd/4.0/ Research Collection School Of Computing and Information Systems eng Institutional Knowledge at Singapore Management University Text generation Vision-and-language Databases and Information Systems Numerical Analysis and Scientific Computing |
institution |
Singapore Management University |
building |
SMU Libraries |
continent |
Asia |
country |
Singapore Singapore |
content_provider |
SMU Libraries |
collection |
InK@SMU |
language |
English |
topic |
Text generation Vision-and-language Databases and Information Systems Numerical Analysis and Scientific Computing |
spellingShingle |
Text generation Vision-and-language Databases and Information Systems Numerical Analysis and Scientific Computing WANG, Hao LIN, Guosheng HOI, Steven C. H. MIAO, Chunyan Decomposing generation networks with structure prediction for recipe generation |
description |
Recipe generation from food images and ingredients is a challenging task, which requires the interpretation of the information from another modality. Different from the image captioning task, where the captions usually have one sentence, cooking instructions contain multiple sentences and have obvious structures. To help the model capture the recipe structure and avoid missing some cooking details, we propose a novel framework: Decomposing Generation Networks (DGN) with structure prediction, to get more structured and complete recipe generation outputs. Specifically, we split each cooking instruction into several phases, and assign different sub-generators to each phase. Our approach includes two novel ideas: (i) learning the recipe structures with the global structure prediction component and (ii) producing recipe phases in the sub-generator output component based on the predicted structure. Extensive experiments on the challenging large-scale Recipe1M dataset validate the effectiveness of our proposed model, which improves the performance over the state-of-the-art results. |
format |
text |
author |
WANG, Hao LIN, Guosheng HOI, Steven C. H. MIAO, Chunyan |
author_facet |
WANG, Hao LIN, Guosheng HOI, Steven C. H. MIAO, Chunyan |
author_sort |
WANG, Hao |
title |
Decomposing generation networks with structure prediction for recipe generation |
title_short |
Decomposing generation networks with structure prediction for recipe generation |
title_full |
Decomposing generation networks with structure prediction for recipe generation |
title_fullStr |
Decomposing generation networks with structure prediction for recipe generation |
title_full_unstemmed |
Decomposing generation networks with structure prediction for recipe generation |
title_sort |
decomposing generation networks with structure prediction for recipe generation |
publisher |
Institutional Knowledge at Singapore Management University |
publishDate |
2022 |
url |
https://ink.library.smu.edu.sg/sis_research/6962 https://ink.library.smu.edu.sg/context/sis_research/article/7965/viewcontent/DecomposingCHRecipe_av.pdf |
_version_ |
1770576166616301568 |