Towards understanding why mask reconstruction pretraining helps in downstream tasks

Towards understanding why mask reconstruction pretraining helps in downstream tasks

For unsupervised pretraining, mask-reconstruction pretraining (MRP) approaches, e.g. MAE (He et al., 2021) and data2vec (Baevski et al., 2022), randomly mask input patches and then reconstruct the pixels or semantic features of these masked patches via an auto-encoder. Then for a downstream task, su...

Saved in:

書目詳細資料
Main Authors:	PAN, Jiachun, ZHOU, Pan, YAN, Shuicheng
格式:	text
語言:	English
出版:	Institutional Knowledge at Singapore Management University 2023
主題:	Graphics and Human Computer Interfaces
在線閱讀:	https://ink.library.smu.edu.sg/sis_research/9022 https://ink.library.smu.edu.sg/context/sis_research/article/10025/viewcontent/2023_ICLR_MAE_Theory.pdf
標簽:	添加標簽沒有標簽, 成為第一個標記此記錄!
機構:	Singapore Management University
語言:	English

相似書籍

Task relation networks
由: LI, Jianshu, et al.
出版: (2019)

Towards understanding convergence and generalization of AdamW
由: ZHOU, Pan, et al.
出版: (2024)

LPT: Long-tailed prompt tuning for image classification
由: DONG, Bowen, et al.
出版: (2023)

Masked diffusion transformer is a strong image synthesizer
由: GAO, Shanghua, et al.
出版: (2023)

Efficient meta learning via minibatch proximal update
由: ZHOU, Pan, et al.
出版: (2019)

EditAnything: Empowering unparalleled flexibility in image editing and generation
由: GAO, Shanghua, et al.
出版: (2023)

Expertise style transfer: A new task towards better communication between experts and laymen
由: CAO, Yixin, et al.
出版: (2020)

Consistent3D: Towards consistent high-fidelity text-to-3D generation with deterministic sampling prior
由: WU, Zike, et al.
出版: (2024)

Position-guided text prompt for vision-language pre-training
由: WANG, Alex Jinpeng, et al.
出版: (2023)

Generalized graph prompt: Toward a unification of pre-training and downstream tasks on graphs
由: YU, Xingtong, et al.
出版: (2024)

InceptionNeXt: When Inception meets ConvNeXt
由: YU, Weihao, et al.
出版: (2024)

Self-promoted supervision for few-shot transformer
由: DONG, Bowen, et al.
出版: (2022)

A theory-driven self-labeling refinement method for contrastive representation learning
由: ZHOU, Pan, et al.
出版: (2021)

Faster first-order methods for stochastic non-convex optimization on Riemannian manifolds
由: ZHOU, Pan, et al.
出版: (2019)

Towards understanding why Lookahead generalizes better than SGD and beyond
由: ZHOU, Pan, et al.
出版: (2021)

Video graph transformer for video question answering
由: XIAO, Junbin, et al.
出版: (2022)

Prototypical contrastive learning of unsupervised representations
由: LI, Junnan, et al.
出版: (2021)

Few-shot learner parameterization by diffusion time-steps
由: YUE, Zhongqi, et al.
出版: (2024)

Motion-based approach for BBC rushes structuring and characterization
由: NGO, Chong-wah, et al.
出版: (2005)

Improving GAN training with probability ratio clipping and sample reweighting
由: WU, Yue, et al.
出版: (2020)

Diffusion time-step curriculum for one image to 3D generation
由: YI, Xuanyu, et al.
出版: (2024)

Graph-evolving meta-learning for low-resource medical dialogue generation
由: LIN, Shuai, et al.
出版: (2021)

DualFormer: Local-global stratified transformer for efficient video recognition
由: LIANG, Yuxuan, et al.
出版: (2022)

VireoJD-MM @ TRECVID 2019: Activities in extended video (ACTEV)
由: HOU, Zhijian, et al.
出版: (2019)

Facial emotion recognition with noisy multi-task annotations
由: ZHANG, S., et al.
出版: (2021)

How knowledge graph and attention help? A qualitative analysis into bag-level relation extraction
由: HU, Zikun, et al.
出版: (2021)

Outlier-robust tensor PCA
由: ZHOU, Pan, et al.
出版: (2016)

Mask-guided deformation adaptive network for human parsing
由: MAO, Aihua, et al.
出版: (2022)

Replay-and-forget-free graph class-incremental learning: A task profiling and prompting approach
由: NIU, Chaoxi, et al.
出版: (2024)

Novel seed selection for multiple objects detection and tracking
由: PAN, Zailiang, et al.
出版: (2004)

Moving-object detection, association, and selection in home videos
由: PAN, Zailiang, et al.
出版: (2007)

Few-shot contrastive transfer learning with pretrained model for masked face verification
由: Weng, Zhenyu, et al.
出版: (2024)

Global context aware convolutions for 3D point cloud understanding
由: ZHANG, Zhiyuan, et al.
出版: (2020)

DAISY filter flow: A generalized discrete approach to dense correspondences
由: YANG, Hongsheng, et al.
出版: (2014)

Causal attention for unbiased visual recognition
由: WANG, Tan, et al.
出版: (2021)

Exploring category-agnostic clusters for open-set domain adaptation
由: PAN, Yingwei, et al.
出版: (2020)

Adversarial meta sampling for multilingual low-resource speech recognition
由: XIAO, Yubei, et al.
出版: (2021)

Wav-BERT: Cooperative acoustic and linguistic representation learning for low-resource speech recognition
由: ZHENG, Guolin, et al.
出版: (2021)

MetaFormer baselines for vision
由: YU, Weihao, et al.
出版: (2023)

PRETRAINED LARGE LANGUAGE MODELS FOR DEFINITION MODELING AND REVERSE DICTIONARY TASKS
由: Nugraha, Irfan