Co-advise: Cross inductive bias distillation

• Main Authors: REN, Sucheng, GAO, Zhengqi, HUA, Tiany, XUE, Zihui, TIAN, Yonglong, HE, Shengfeng, ZHAO, Hang
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2022
• Subjects: Adversarial attack and defense; Distillation method; Inductive bias; Performance; Representation learning; Size models; Teacher models; Teachers'; Databases and Information Systems
• Online Access: https://ink.library.smu.edu.sg/sis_research/8538; https://ink.library.smu.edu.sg/context/sis_research/article/9541/viewcontent/Co_Advise__Cross_Inductive_Bias_Distillation.pdf
• Institution: Singapore Management University

The inductive bias of vision transformers is more relaxed that cannot work well with insufficient data. Knowledge distillation is thus introduced to assist the training of transformers. Unlike previous works, where merely heavy convolution-based teachers are provided, in this paper, we delve into th...