A decoupled learning framework for contrastive learning

• Main Author: Xu, Yicheng
• Other Authors: Lin Zhiping
• Format: Thesis-Master by Coursework
• Language: English
• Published: Nanyang Technological University 2022
• Subjects: Engineering::Electrical and electronic engineering::Computer hardware, software and systems
• Online Access: https://hdl.handle.net/10356/163711
• Institution: Nanyang Technological University

Contrastive Learning (CL) has attracted much attention in recent years because various self-supervised models based on CL achieve comparable performance to supervised models. Nevertheless, most CL frameworks require large batch size during the training progress for taking more negative samples into account to boost the performance. Meanwhile, the large model size limits the training batch size under fixed device memory. To solve this problem, we propose a Decoupled Updating Contrastive Learning (DUCL) framework 1) to divide a single model into pieces to shrink the model size on each accelerator device and 2) to decouple every batch in CL for memory saving. The combination of both approaches enables a larger negative sample space for contrastive learning models to achieve better performance. As a result, we prove the effectiveness of large batch size and save the memory to a maximum of 43% in our experiments. By incorporating our learning method, the contrastive learning model can be trained with a larger negative sample space thus improving its performance without making any change for the model structure.