Towards understanding why mask reconstruction pretraining helps in downstream tasks

• المؤلفون الرئيسيون: PAN, Jiachun, ZHOU, Pan, YAN, Shuicheng
• التنسيق: text
• اللغة: English
• منشور في: Institutional Knowledge at Singapore Management University 2023
• الموضوعات: Graphics and Human Computer Interfaces
• الوصول للمادة أونلاين: https://ink.library.smu.edu.sg/sis_research/9022; https://ink.library.smu.edu.sg/context/sis_research/article/10025/viewcontent/2023_ICLR_MAE_Theory.pdf

For unsupervised pretraining, mask-reconstruction pretraining (MRP) approaches, e.g. MAE (He et al., 2021) and data2vec (Baevski et al., 2022), randomly mask input patches and then reconstruct the pixels or semantic features of these masked patches via an auto-encoder. Then for a downstream task, su...