On-the-fly knowledge distillation model for sentence embedding

In this dissertation, we run experimental study to investigate the performance of sentence embedding using an on-the-fly knowledge distillation model based on DistillCSE framework. This model utilizes SimCSE as the initial teacher model. After a certain number of training steps, it caches an interm...

Full description

Saved in:

Bibliographic Details
Main Author:	Zhu, Xuchun
Other Authors:	Lihui Chen
Format:	Thesis-Master by Coursework
Language:	English
Published:	Nanyang Technological University 2024
Subjects:	Computer and Information Science On-the-fly model Knowledge distillation Sentence embeddings SimCSE DistillCSE
Online Access:	https://hdl.handle.net/10356/174236
Tags:	Add Tag No Tags, Be the first to tag this record!
Institution:	Nanyang Technological University
Language:	English

id	sg-ntu-dr.10356-174236
record_format	dspace
spelling	sg-ntu-dr.10356-1742362024-03-29T15:43:33Z On-the-fly knowledge distillation model for sentence embedding Zhu, Xuchun Lihui Chen School of Electrical and Electronic Engineering ELHCHEN@ntu.edu.sg Computer and Information Science On-the-fly model Knowledge distillation Sentence embeddings SimCSE DistillCSE In this dissertation, we run experimental study to investigate the performance of sentence embedding using an on-the-fly knowledge distillation model based on DistillCSE framework. This model utilizes SimCSE as the initial teacher model. After a certain number of training steps, it caches an intermediate model and employs it as a new teacher model for knowledge distillation. This process is repeated several times to obtain the desired on-the-fly knowledge distilled student model. This model employs a novel approach to knowledge distillation, potentially offering advantages such as reducing training time and achieving performance close to the original teacher model. In some cases, after fine-tuning, it may even surpass the performance of the original teacher model for specific tasks. Master's degree 2024-03-25T01:05:10Z 2024-03-25T01:05:10Z 2024 Thesis-Master by Coursework Zhu, X. (2024). On-the-fly knowledge distillation model for sentence embedding. Master's thesis, Nanyang Technological University, Singapore. https://hdl.handle.net/10356/174236 https://hdl.handle.net/10356/174236 en D-258-22231-05829 application/pdf Nanyang Technological University
institution	Nanyang Technological University
building	NTU Library
continent	Asia
country	Singapore Singapore
content_provider	NTU Library
collection	DR-NTU
language	English
topic	Computer and Information Science On-the-fly model Knowledge distillation Sentence embeddings SimCSE DistillCSE
spellingShingle	Computer and Information Science On-the-fly model Knowledge distillation Sentence embeddings SimCSE DistillCSE Zhu, Xuchun On-the-fly knowledge distillation model for sentence embedding
description	In this dissertation, we run experimental study to investigate the performance of sentence embedding using an on-the-fly knowledge distillation model based on DistillCSE framework. This model utilizes SimCSE as the initial teacher model. After a certain number of training steps, it caches an intermediate model and employs it as a new teacher model for knowledge distillation. This process is repeated several times to obtain the desired on-the-fly knowledge distilled student model. This model employs a novel approach to knowledge distillation, potentially offering advantages such as reducing training time and achieving performance close to the original teacher model. In some cases, after fine-tuning, it may even surpass the performance of the original teacher model for specific tasks.
author2	Lihui Chen
author_facet	Lihui Chen Zhu, Xuchun
format	Thesis-Master by Coursework
author	Zhu, Xuchun
author_sort	Zhu, Xuchun
title	On-the-fly knowledge distillation model for sentence embedding
title_short	On-the-fly knowledge distillation model for sentence embedding
title_full	On-the-fly knowledge distillation model for sentence embedding
title_fullStr	On-the-fly knowledge distillation model for sentence embedding
title_full_unstemmed	On-the-fly knowledge distillation model for sentence embedding
title_sort	on-the-fly knowledge distillation model for sentence embedding
publisher	Nanyang Technological University
publishDate	2024
url	https://hdl.handle.net/10356/174236
_version_	1795302091709743104

On-the-fly knowledge distillation model for sentence embedding

Similar Items