Knowledge Distillation with Relative Representations for Image Representation Learning

• Main Authors: Ramos, Patrick, Alampay, Raphael, Abu, Patricia Angela R
• Format: text
• Published: Archīum Ateneo 2023
• Subjects: Knowledge distillation; Latent space; Relative representations; Computer Engineering; Electrical and Computer Engineering; Engineering
• Online Access: https://archium.ateneo.edu/discs-faculty-pubs/387; https://doi.org/10.1007/978-3-031-41630-9_14
• Institution: Ateneo De Manila University

Relative representations allow the alignment of latent spaces which embed data in extrinsically different manners but with similar relative distances between data points. This ability to compare different latent spaces for the same input lends itself to knowledge distillation techniques. We explore...