On-the-fly knowledge distillation model for sentence embedding

• Main Author: Zhu, Xuchun
• Other Authors: Lihui Chen
• Format: Thesis-Master by Coursework
• Language: English
• Published: Nanyang Technological University 2024
• Subjects: Computer and Information Science; On-the-fly model; Knowledge distillation; Sentence embeddings; SimCSE; DistillCSE
• Online Access: https://hdl.handle.net/10356/174236
• Institution: Nanyang Technological University

In this dissertation, we run experimental study to investigate the performance of sentence embedding using an on-the-fly knowledge distillation model based on DistillCSE framework. This model utilizes SimCSE as the initial teacher model. After a certain number of training steps, it caches an interm...