Towards understanding why mask reconstruction pretraining helps in downstream tasks

• Main Authors: PAN, Jiachun, ZHOU, Pan, YAN, Shuicheng
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2023
• Subjects: Graphics and Human Computer Interfaces
• Online Access: https://ink.library.smu.edu.sg/sis_research/9022; https://ink.library.smu.edu.sg/context/sis_research/article/10025/viewcontent/2023_ICLR_MAE_Theory.pdf
• Institution: Singapore Management University

For unsupervised pretraining, mask-reconstruction pretraining (MRP) approaches, e.g. MAE (He et al., 2021) and data2vec (Baevski et al., 2022), randomly mask input patches and then reconstruct the pixels or semantic features of these masked patches via an auto-encoder. Then for a downstream task, su...