Declaration-based prompt tuning for visual question answering

In recent years, the pre-training-then-fine-tuning paradigm has yielded immense success on a wide spectrum of cross-modal tasks, such as visual question answering (VQA), in which a visual-language (VL) model is first optimized via self-supervised task objectives, e.g., masked language modeling (MLM)...

Full description

Saved in:
Bibliographic Details
Main Authors: LIU, Yuhang, WEI, Wei, ZHU, Feida
Format: text
Language:English
Published: Institutional Knowledge at Singapore Management University 2022
Subjects:
Online Access:https://ink.library.smu.edu.sg/sis_research/7752
https://ink.library.smu.edu.sg/context/sis_research/article/8755/viewcontent/declaration.pdf
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Singapore Management University
Language: English
id sg-smu-ink.sis_research-8755
record_format dspace
spelling sg-smu-ink.sis_research-87552023-01-19T10:13:03Z Declaration-based prompt tuning for visual question answering LIU, Yuhang WEI, Wei ZHU, Feida ZHU, Feida In recent years, the pre-training-then-fine-tuning paradigm has yielded immense success on a wide spectrum of cross-modal tasks, such as visual question answering (VQA), in which a visual-language (VL) model is first optimized via self-supervised task objectives, e.g., masked language modeling (MLM) and image-text matching (ITM), and then fine-tuned to adapt to downstream task (e.g., VQA) via a brand-new objective function, e.g., answer prediction. However, the inconsistency of the objective forms not only severely limits the generalization of pre-trained VL models to downstream tasks, but also requires a large amount of labeled data for fine-tuning. To alleviate the problem, we propose an innovative VL fine-tuning paradigm (named Declaration-based Prompt Tuning, abbreviated as DPT), which fine-tunes the model for downstream VQA using the pre-training objectives, boosting the effective adaptation of pre-trained models to the downstream task. Specifically, DPT reformulates the VQA task via (1) textual adaptation, which converts the given questions into declarative sentence form for prompt-tuning, and (2) task adaptation, which optimizes the objective function of VQA problem in the manner of pre-training phase. Experimental results on GQA dataset show that DPT outperforms the fine-tuned counterpart by a large margin regarding accuracy in both fully-supervised (2.68%) and zero-shot/fewshot (over 31%) settings. All the data and codes will be available to facilitate future research. 2022-07-01T07:00:00Z text application/pdf https://ink.library.smu.edu.sg/sis_research/7752 info:doi/10.24963/ijcai.2022/453 https://ink.library.smu.edu.sg/context/sis_research/article/8755/viewcontent/declaration.pdf http://creativecommons.org/licenses/by-nc-nd/4.0/ Research Collection School Of Computing and Information Systems eng Institutional Knowledge at Singapore Management University Machine Learning: Multi-modal learning Computer Vision: Transfer low-shot semi- and un- supervised learning Computer Vision: Vision and language Natural Language Processing: Question Answering Databases and Information Systems
institution Singapore Management University
building SMU Libraries
continent Asia
country Singapore
Singapore
content_provider SMU Libraries
collection InK@SMU
language English
topic Machine Learning: Multi-modal learning
Computer Vision: Transfer
low-shot
semi- and un- supervised learning
Computer Vision: Vision and language
Natural Language Processing: Question Answering
Databases and Information Systems
spellingShingle Machine Learning: Multi-modal learning
Computer Vision: Transfer
low-shot
semi- and un- supervised learning
Computer Vision: Vision and language
Natural Language Processing: Question Answering
Databases and Information Systems
LIU, Yuhang
WEI, Wei
ZHU, Feida
ZHU, Feida
Declaration-based prompt tuning for visual question answering
description In recent years, the pre-training-then-fine-tuning paradigm has yielded immense success on a wide spectrum of cross-modal tasks, such as visual question answering (VQA), in which a visual-language (VL) model is first optimized via self-supervised task objectives, e.g., masked language modeling (MLM) and image-text matching (ITM), and then fine-tuned to adapt to downstream task (e.g., VQA) via a brand-new objective function, e.g., answer prediction. However, the inconsistency of the objective forms not only severely limits the generalization of pre-trained VL models to downstream tasks, but also requires a large amount of labeled data for fine-tuning. To alleviate the problem, we propose an innovative VL fine-tuning paradigm (named Declaration-based Prompt Tuning, abbreviated as DPT), which fine-tunes the model for downstream VQA using the pre-training objectives, boosting the effective adaptation of pre-trained models to the downstream task. Specifically, DPT reformulates the VQA task via (1) textual adaptation, which converts the given questions into declarative sentence form for prompt-tuning, and (2) task adaptation, which optimizes the objective function of VQA problem in the manner of pre-training phase. Experimental results on GQA dataset show that DPT outperforms the fine-tuned counterpart by a large margin regarding accuracy in both fully-supervised (2.68%) and zero-shot/fewshot (over 31%) settings. All the data and codes will be available to facilitate future research.
format text
author LIU, Yuhang
WEI, Wei
ZHU, Feida
ZHU, Feida
author_facet LIU, Yuhang
WEI, Wei
ZHU, Feida
ZHU, Feida
author_sort LIU, Yuhang
title Declaration-based prompt tuning for visual question answering
title_short Declaration-based prompt tuning for visual question answering
title_full Declaration-based prompt tuning for visual question answering
title_fullStr Declaration-based prompt tuning for visual question answering
title_full_unstemmed Declaration-based prompt tuning for visual question answering
title_sort declaration-based prompt tuning for visual question answering
publisher Institutional Knowledge at Singapore Management University
publishDate 2022
url https://ink.library.smu.edu.sg/sis_research/7752
https://ink.library.smu.edu.sg/context/sis_research/article/8755/viewcontent/declaration.pdf
_version_ 1770576434810585088