Knowledge generation for zero-shot knowledge-based VQA
Previous solutions to knowledge-based visual question answering (K-VQA) retrieve knowledge from external knowledge bases and use supervised learning to train the K-VQA model. Recently pre-trained LLMs have been used as both a knowledge source and a zero-shot QA model for K-VQA and demonstrated promi...
Saved in:
Main Authors: | , |
---|---|
Format: | text |
Language: | English |
Published: |
Institutional Knowledge at Singapore Management University
2024
|
Subjects: | |
Online Access: | https://ink.library.smu.edu.sg/sis_research/8726 https://ink.library.smu.edu.sg/context/sis_research/article/9729/viewcontent/2024.findings_eacl.36_pvoa.pdf |
Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
Institution: | Singapore Management University |
Language: | English |
id |
sg-smu-ink.sis_research-9729 |
---|---|
record_format |
dspace |
spelling |
sg-smu-ink.sis_research-97292024-04-18T07:34:44Z Knowledge generation for zero-shot knowledge-based VQA CAO, Rui JIANG, Jing Previous solutions to knowledge-based visual question answering (K-VQA) retrieve knowledge from external knowledge bases and use supervised learning to train the K-VQA model. Recently pre-trained LLMs have been used as both a knowledge source and a zero-shot QA model for K-VQA and demonstrated promising results. However, these recent methods do not explicitly show the knowledge needed to answer the questions and thus lack interpretability. Inspired by recent work on knowledge generation from LLMs for text-based QA, in this work we propose and test a similar knowledge-generation-based K-VQA method, which first generates knowledge from an LLM and then incorporates the generated knowledge for K-VQA in a zero-shot manner. We evaluate our method on two K-VQA benchmarks and found that our method performs better than previous zero-shot K-VQA methods and our generated knowledge is generally relevant and helpful. 2024-03-01T08:00:00Z text application/pdf https://ink.library.smu.edu.sg/sis_research/8726 https://ink.library.smu.edu.sg/context/sis_research/article/9729/viewcontent/2024.findings_eacl.36_pvoa.pdf http://creativecommons.org/licenses/by-nc-nd/4.0/ Research Collection School Of Computing and Information Systems eng Institutional Knowledge at Singapore Management University Artificial Intelligence and Robotics Numerical Analysis and Scientific Computing |
institution |
Singapore Management University |
building |
SMU Libraries |
continent |
Asia |
country |
Singapore Singapore |
content_provider |
SMU Libraries |
collection |
InK@SMU |
language |
English |
topic |
Artificial Intelligence and Robotics Numerical Analysis and Scientific Computing |
spellingShingle |
Artificial Intelligence and Robotics Numerical Analysis and Scientific Computing CAO, Rui JIANG, Jing Knowledge generation for zero-shot knowledge-based VQA |
description |
Previous solutions to knowledge-based visual question answering (K-VQA) retrieve knowledge from external knowledge bases and use supervised learning to train the K-VQA model. Recently pre-trained LLMs have been used as both a knowledge source and a zero-shot QA model for K-VQA and demonstrated promising results. However, these recent methods do not explicitly show the knowledge needed to answer the questions and thus lack interpretability. Inspired by recent work on knowledge generation from LLMs for text-based QA, in this work we propose and test a similar knowledge-generation-based K-VQA method, which first generates knowledge from an LLM and then incorporates the generated knowledge for K-VQA in a zero-shot manner. We evaluate our method on two K-VQA benchmarks and found that our method performs better than previous zero-shot K-VQA methods and our generated knowledge is generally relevant and helpful. |
format |
text |
author |
CAO, Rui JIANG, Jing |
author_facet |
CAO, Rui JIANG, Jing |
author_sort |
CAO, Rui |
title |
Knowledge generation for zero-shot knowledge-based VQA |
title_short |
Knowledge generation for zero-shot knowledge-based VQA |
title_full |
Knowledge generation for zero-shot knowledge-based VQA |
title_fullStr |
Knowledge generation for zero-shot knowledge-based VQA |
title_full_unstemmed |
Knowledge generation for zero-shot knowledge-based VQA |
title_sort |
knowledge generation for zero-shot knowledge-based vqa |
publisher |
Institutional Knowledge at Singapore Management University |
publishDate |
2024 |
url |
https://ink.library.smu.edu.sg/sis_research/8726 https://ink.library.smu.edu.sg/context/sis_research/article/9729/viewcontent/2024.findings_eacl.36_pvoa.pdf |
_version_ |
1814047494888226816 |