Question type-aware debiasing for test-time visual question answering model adaptation

In Visual Question Answering (VQA), addressing language prior bias, where models excessively rely on superficial correlations between questions and answers, is crucial. This issue becomes more pronounced in real-world applications with diverse domains and varied question-answer distributions during...

Full description

Saved in:
Bibliographic Details
Main Authors: LIU, Jin, XIE, Jialong, ZHOU, Fengyu, HE, Shengfeng
Format: text
Language:English
Published: Institutional Knowledge at Singapore Management University 2024
Subjects:
Online Access:https://ink.library.smu.edu.sg/sis_research/9804
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Singapore Management University
Language: English
id sg-smu-ink.sis_research-10804
record_format dspace
spelling sg-smu-ink.sis_research-108042024-12-12T09:00:03Z Question type-aware debiasing for test-time visual question answering model adaptation LIU, Jin XIE, Jialong ZHOU, Fengyu HE, Shengfeng In Visual Question Answering (VQA), addressing language prior bias, where models excessively rely on superficial correlations between questions and answers, is crucial. This issue becomes more pronounced in real-world applications with diverse domains and varied question-answer distributions during testing. To tackle this challenge, Test-time Adaptation (TTA) has emerged, allowing pre-trained VQA models to adapt using unlabeled test samples. Current state-of-the-art models select reliable test samples based on fixed entropy thresholds and employ self-supervised debiasing techniques. However, these methods struggle with diverse answer spaces linked to different question types and may fail to identify biased samples that still leverage relevant visual context. In this paper, we propose Question type-guided Entropy Minimization and Debiasing (QED) as a solution for test-time VQA model adaptation. Our approach involves adaptive entropy minimization based on question types to improve the identification of fine-grained and unreliable samples. Additionally, we generate negative samples for each test sample and label them as biased if their answer entropy change rate significantly differs from positive test samples, subsequently removing them. We evaluate our approach on two public benchmarks, VQA-CP v2, and VQA-CP v1, and achieve new state-of-the-art results, with overall accuracy rates of 48.13% and 46.18%, respectively. 2024-06-05T07:00:00Z text https://ink.library.smu.edu.sg/sis_research/9804 info:doi/10.1109/TCSVT.2024.3410041 Research Collection School Of Computing and Information Systems eng Institutional Knowledge at Singapore Management University Test-time adaptation visual question answering language debiasing Artificial Intelligence and Robotics Graphics and Human Computer Interfaces
institution Singapore Management University
building SMU Libraries
continent Asia
country Singapore
Singapore
content_provider SMU Libraries
collection InK@SMU
language English
topic Test-time adaptation
visual question answering
language debiasing
Artificial Intelligence and Robotics
Graphics and Human Computer Interfaces
spellingShingle Test-time adaptation
visual question answering
language debiasing
Artificial Intelligence and Robotics
Graphics and Human Computer Interfaces
LIU, Jin
XIE, Jialong
ZHOU, Fengyu
HE, Shengfeng
Question type-aware debiasing for test-time visual question answering model adaptation
description In Visual Question Answering (VQA), addressing language prior bias, where models excessively rely on superficial correlations between questions and answers, is crucial. This issue becomes more pronounced in real-world applications with diverse domains and varied question-answer distributions during testing. To tackle this challenge, Test-time Adaptation (TTA) has emerged, allowing pre-trained VQA models to adapt using unlabeled test samples. Current state-of-the-art models select reliable test samples based on fixed entropy thresholds and employ self-supervised debiasing techniques. However, these methods struggle with diverse answer spaces linked to different question types and may fail to identify biased samples that still leverage relevant visual context. In this paper, we propose Question type-guided Entropy Minimization and Debiasing (QED) as a solution for test-time VQA model adaptation. Our approach involves adaptive entropy minimization based on question types to improve the identification of fine-grained and unreliable samples. Additionally, we generate negative samples for each test sample and label them as biased if their answer entropy change rate significantly differs from positive test samples, subsequently removing them. We evaluate our approach on two public benchmarks, VQA-CP v2, and VQA-CP v1, and achieve new state-of-the-art results, with overall accuracy rates of 48.13% and 46.18%, respectively.
format text
author LIU, Jin
XIE, Jialong
ZHOU, Fengyu
HE, Shengfeng
author_facet LIU, Jin
XIE, Jialong
ZHOU, Fengyu
HE, Shengfeng
author_sort LIU, Jin
title Question type-aware debiasing for test-time visual question answering model adaptation
title_short Question type-aware debiasing for test-time visual question answering model adaptation
title_full Question type-aware debiasing for test-time visual question answering model adaptation
title_fullStr Question type-aware debiasing for test-time visual question answering model adaptation
title_full_unstemmed Question type-aware debiasing for test-time visual question answering model adaptation
title_sort question type-aware debiasing for test-time visual question answering model adaptation
publisher Institutional Knowledge at Singapore Management University
publishDate 2024
url https://ink.library.smu.edu.sg/sis_research/9804
_version_ 1819113143236820992