Prototypical contrastive learning of unsupervised representations
This paper presents Prototypical Contrastive Learning (PCL), an unsupervised representation learning method that bridges contrastive learning with clustering. PCL not only learns low-level features for the task of instance discrimination, but more importantly, it encodes semantic structures discover...
Saved in:
Main Authors: | , , , |
---|---|
Format: | text |
Language: | English |
Published: |
Institutional Knowledge at Singapore Management University
2021
|
Subjects: | |
Online Access: | https://ink.library.smu.edu.sg/sis_research/8993 https://ink.library.smu.edu.sg/context/sis_research/article/9996/viewcontent/2021_ICLR_PCL.pdf |
Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
Institution: | Singapore Management University |
Language: | English |
id |
sg-smu-ink.sis_research-9996 |
---|---|
record_format |
dspace |
spelling |
sg-smu-ink.sis_research-99962024-07-25T08:25:31Z Prototypical contrastive learning of unsupervised representations LI, Junnan ZHOU, Pan XIONG, Caiming HOI, Steven C. H. This paper presents Prototypical Contrastive Learning (PCL), an unsupervised representation learning method that bridges contrastive learning with clustering. PCL not only learns low-level features for the task of instance discrimination, but more importantly, it encodes semantic structures discovered by clustering into the learned embedding space. Specifically, we introduce prototypes as latent variables to help find the maximum-likelihood estimation of the network parameters in an Expectation-Maximization framework. We iteratively perform E-step as finding the distribution of prototypes via clustering and M-step as optimizing the network via contrastive learning. We propose ProtoNCE loss, a generalized version of the InfoNCE loss for contrastive learning, which encourages representations to be closer to their assigned prototypes. PCL outperforms state-of-the-art instance-wise contrastive learning methods on multiple benchmarks with substantial improvement in low-resource transfer learning. Code and pretrained models are available at https://github.com/salesforce/PCL. 2021-05-01T07:00:00Z text application/pdf https://ink.library.smu.edu.sg/sis_research/8993 https://ink.library.smu.edu.sg/context/sis_research/article/9996/viewcontent/2021_ICLR_PCL.pdf http://creativecommons.org/licenses/by-nc-nd/4.0/ Research Collection School Of Computing and Information Systems eng Institutional Knowledge at Singapore Management University Graphics and Human Computer Interfaces |
institution |
Singapore Management University |
building |
SMU Libraries |
continent |
Asia |
country |
Singapore Singapore |
content_provider |
SMU Libraries |
collection |
InK@SMU |
language |
English |
topic |
Graphics and Human Computer Interfaces |
spellingShingle |
Graphics and Human Computer Interfaces LI, Junnan ZHOU, Pan XIONG, Caiming HOI, Steven C. H. Prototypical contrastive learning of unsupervised representations |
description |
This paper presents Prototypical Contrastive Learning (PCL), an unsupervised representation learning method that bridges contrastive learning with clustering. PCL not only learns low-level features for the task of instance discrimination, but more importantly, it encodes semantic structures discovered by clustering into the learned embedding space. Specifically, we introduce prototypes as latent variables to help find the maximum-likelihood estimation of the network parameters in an Expectation-Maximization framework. We iteratively perform E-step as finding the distribution of prototypes via clustering and M-step as optimizing the network via contrastive learning. We propose ProtoNCE loss, a generalized version of the InfoNCE loss for contrastive learning, which encourages representations to be closer to their assigned prototypes. PCL outperforms state-of-the-art instance-wise contrastive learning methods on multiple benchmarks with substantial improvement in low-resource transfer learning. Code and pretrained models are available at https://github.com/salesforce/PCL. |
format |
text |
author |
LI, Junnan ZHOU, Pan XIONG, Caiming HOI, Steven C. H. |
author_facet |
LI, Junnan ZHOU, Pan XIONG, Caiming HOI, Steven C. H. |
author_sort |
LI, Junnan |
title |
Prototypical contrastive learning of unsupervised representations |
title_short |
Prototypical contrastive learning of unsupervised representations |
title_full |
Prototypical contrastive learning of unsupervised representations |
title_fullStr |
Prototypical contrastive learning of unsupervised representations |
title_full_unstemmed |
Prototypical contrastive learning of unsupervised representations |
title_sort |
prototypical contrastive learning of unsupervised representations |
publisher |
Institutional Knowledge at Singapore Management University |
publishDate |
2021 |
url |
https://ink.library.smu.edu.sg/sis_research/8993 https://ink.library.smu.edu.sg/context/sis_research/article/9996/viewcontent/2021_ICLR_PCL.pdf |
_version_ |
1814047703057825792 |