Counterfactual zero-shot and open-set visual recognition
We present a novel counterfactual framework for both Zero-Shot Learning (ZSL) and Open-Set Recognition (OSR), whose common challenge is generalizing to the unseen-classes by only training on the seen-classes. Our idea stems from the observation that the generated samples for unseen-classes are often...
Saved in:
Main Authors: | , , , , |
---|---|
Format: | text |
Language: | English |
Published: |
Institutional Knowledge at Singapore Management University
2021
|
Subjects: | |
Online Access: | https://ink.library.smu.edu.sg/sis_research/6120 https://ink.library.smu.edu.sg/context/sis_research/article/7123/viewcontent/counterfactual_zsl_openset_CVPR2021.pdf |
Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
Institution: | Singapore Management University |
Language: | English |
id |
sg-smu-ink.sis_research-7123 |
---|---|
record_format |
dspace |
spelling |
sg-smu-ink.sis_research-71232022-05-18T07:07:42Z Counterfactual zero-shot and open-set visual recognition YUE, Zhongqi WANG, Tan SUN, Qianru HUA, Xian-Sheng ZHANG, Hanwang We present a novel counterfactual framework for both Zero-Shot Learning (ZSL) and Open-Set Recognition (OSR), whose common challenge is generalizing to the unseen-classes by only training on the seen-classes. Our idea stems from the observation that the generated samples for unseen-classes are often out of the true distribution, which causes severe recognition rate imbalance between the seen-class (high) and unseen-class (low). We show that the key reason is that the generation is not Counterfactual Faithful, and thus we propose a faithful one, whose generation is from the sample-specific counterfactual question: What would the sample look like, if we set its class attribute to a certain class, while keeping its sample attribute unchanged? Thanks to the faithfulness, we can apply the Consistency Rule to perform unseen/seen binary classification, by asking: Would its counterfactual still look like itself? If ``yes'', the sample is from a certain class, and ``no'' otherwise. Through extensive experiments on ZSL and OSR, we demonstrate that our framework effectively mitigates the seen/unseen imbalance and hence significantly improves the overall performance. Note that this framework is orthogonal to existing methods, thus, it can serve as a new baseline to evaluate how ZSL/OSR models generalize. 2021-06-01T07:00:00Z text application/pdf https://ink.library.smu.edu.sg/sis_research/6120 info:doi/10.1109/CVPR46437.2021.01515 https://ink.library.smu.edu.sg/context/sis_research/article/7123/viewcontent/counterfactual_zsl_openset_CVPR2021.pdf http://creativecommons.org/licenses/by-nc-nd/4.0/ Research Collection School Of Computing and Information Systems eng Institutional Knowledge at Singapore Management University Training visualization computer vision codes pattern recognition Artificial Intelligence and Robotics |
institution |
Singapore Management University |
building |
SMU Libraries |
continent |
Asia |
country |
Singapore Singapore |
content_provider |
SMU Libraries |
collection |
InK@SMU |
language |
English |
topic |
Training visualization computer vision codes pattern recognition Artificial Intelligence and Robotics |
spellingShingle |
Training visualization computer vision codes pattern recognition Artificial Intelligence and Robotics YUE, Zhongqi WANG, Tan SUN, Qianru HUA, Xian-Sheng ZHANG, Hanwang Counterfactual zero-shot and open-set visual recognition |
description |
We present a novel counterfactual framework for both Zero-Shot Learning (ZSL) and Open-Set Recognition (OSR), whose common challenge is generalizing to the unseen-classes by only training on the seen-classes. Our idea stems from the observation that the generated samples for unseen-classes are often out of the true distribution, which causes severe recognition rate imbalance between the seen-class (high) and unseen-class (low). We show that the key reason is that the generation is not Counterfactual Faithful, and thus we propose a faithful one, whose generation is from the sample-specific counterfactual question: What would the sample look like, if we set its class attribute to a certain class, while keeping its sample attribute unchanged? Thanks to the faithfulness, we can apply the Consistency Rule to perform unseen/seen binary classification, by asking: Would its counterfactual still look like itself? If ``yes'', the sample is from a certain class, and ``no'' otherwise. Through extensive experiments on ZSL and OSR, we demonstrate that our framework effectively mitigates the seen/unseen imbalance and hence significantly improves the overall performance. Note that this framework is orthogonal to existing methods, thus, it can serve as a new baseline to evaluate how ZSL/OSR models generalize. |
format |
text |
author |
YUE, Zhongqi WANG, Tan SUN, Qianru HUA, Xian-Sheng ZHANG, Hanwang |
author_facet |
YUE, Zhongqi WANG, Tan SUN, Qianru HUA, Xian-Sheng ZHANG, Hanwang |
author_sort |
YUE, Zhongqi |
title |
Counterfactual zero-shot and open-set visual recognition |
title_short |
Counterfactual zero-shot and open-set visual recognition |
title_full |
Counterfactual zero-shot and open-set visual recognition |
title_fullStr |
Counterfactual zero-shot and open-set visual recognition |
title_full_unstemmed |
Counterfactual zero-shot and open-set visual recognition |
title_sort |
counterfactual zero-shot and open-set visual recognition |
publisher |
Institutional Knowledge at Singapore Management University |
publishDate |
2021 |
url |
https://ink.library.smu.edu.sg/sis_research/6120 https://ink.library.smu.edu.sg/context/sis_research/article/7123/viewcontent/counterfactual_zsl_openset_CVPR2021.pdf |
_version_ |
1770575832981438464 |