Knowledge Distillation with Relative Representations for Image Representation Learning

• Main Authors: Ramos, Patrick, Alampay, Raphael, Abu, Patricia Angela R
• Format: text
• Published: Archīum Ateneo 2023
• Subjects: Knowledge distillation; Latent space; Relative representations; Computer Engineering; Electrical and Computer Engineering; Engineering
• Online Access: https://archium.ateneo.edu/discs-faculty-pubs/387; https://doi.org/10.1007/978-3-031-41630-9_14
• Institution: Ateneo De Manila University

Relative representations allow the alignment of latent spaces which embed data in extrinsically different manners but with similar relative distances between data points. This ability to compare different latent spaces for the same input lends itself to knowledge distillation techniques. We explore the applicability of relative representations to knowledge distillation by training a student model such that the relative representations of its outputs match the relative representations of the outputs of a teacher model. We test our Relative Representation Knowledge Distillation (RRKD) scheme on supervised and self-supervised image representation learning with MNIST and show that an encoder can be compressed to 47.71% of its original size while maintaining 91.92% of its full performance. We demonstrate that RRKD is competitive with or outperforms other relation-based distillation schemes in traditional distillation setups (CIFAR-10, CIFAR-100, SVHN) and in a transfer learning setting (Stanford Cars, Oxford-IIIT Pets, Oxford Flowers-102). Our results indicate that relative representations are an effective signal for knowledge distillation. Code is made available at https://github.com/Ramos-Ramos/rrkd.