Equivariance and invariance inductive bias for learning from insufficient data

• Main Authors: WANG, Tan, SUN, Qianru, PRANATA, Sugiri, JAYASHREE, Karlekar, ZHANG, Hanwang
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2022
• Subjects: Inductive Bias; Equivariance; Invariant Risk Minimization; Databases and Information Systems; Graphics and Human Computer Interfaces; Numerical Analysis and Scientific Computing
• Online Access: https://ink.library.smu.edu.sg/sis_research/7513; https://ink.library.smu.edu.sg/context/sis_research/article/8516/viewcontent/ECCV2022_Vipriors_WangTan.pdf
• Institution: Singapore Management University

We are interested in learning robust models from insufficient data, without the need for any externally pre-trained model checkpoints. First, compared to sufficient data, we show why insufficient data renders the model more easily biased to the limited training environments that are usually different from testing. For example, if all the training "swan" samples are "white", the model may wrongly use the "white" environment to represent the intrinsic class "swan". Then, we justify that equivariance inductive bias can retain the class feature while invariance inductive bias can remove the environmental feature, leaving only the class feature that generalizes to any testing environmental changes. To impose them on learning, for equivariance, we demonstrate that any off-the-shelf contrastive-based self-supervised feature learning method can be deployed; for invariance, we propose a class-wise invariant risk minimization (IRM) that efficiently tackles the challenge of missing environmental annotation in conventional IRM. State-of-the-art experimental results on real-world visual benchmarks (NICO and VIPriors ImageNet) validate the great potential of the two inductive biases in reducing training data and parameters significantly.