Counterfactual zero-shot and open-set visual recognition

• Main Authors: YUE, Zhongqi, WANG, Tan, SUN, Qianru, HUA, Xian-Sheng, ZHANG, Hanwang
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2021
• Subjects: Training; visualization; computer vision; codes; pattern recognition; Artificial Intelligence and Robotics
• Online Access: https://ink.library.smu.edu.sg/sis_research/6120; https://ink.library.smu.edu.sg/context/sis_research/article/7123/viewcontent/counterfactual_zsl_openset_CVPR2021.pdf
• Institution: Singapore Management University

We present a novel counterfactual framework for both Zero-Shot Learning (ZSL) and Open-Set Recognition (OSR), whose common challenge is generalizing to the unseen-classes by only training on the seen-classes. Our idea stems from the observation that the generated samples for unseen-classes are often out of the true distribution, which causes severe recognition rate imbalance between the seen-class (high) and unseen-class (low). We show that the key reason is that the generation is not Counterfactual Faithful, and thus we propose a faithful one, whose generation is from the sample-specific counterfactual question: What would the sample look like, if we set its class attribute to a certain class, while keeping its sample attribute unchanged? Thanks to the faithfulness, we can apply the Consistency Rule to perform unseen/seen binary classification, by asking: Would its counterfactual still look like itself? If ``yes'', the sample is from a certain class, and ``no'' otherwise. Through extensive experiments on ZSL and OSR, we demonstrate that our framework effectively mitigates the seen/unseen imbalance and hence significantly improves the overall performance. Note that this framework is orthogonal to existing methods, thus, it can serve as a new baseline to evaluate how ZSL/OSR models generalize.