Large language model enhanced with prompt-based vanilla distillation for sentence embeddings

Large language model enhanced with prompt-based vanilla distillation for sentence embeddings

In this dissertation, the prompt-based method PromptEOL is used to train the opt- 2.7b model with the Parameter-Efficient Fine-Tuning method to reduce the number of training parameters and GPU memory usage. Then the opt-2.7b-lora model is used as the teacher model to train the student model under...

Full description

Saved in:
Bibliographic Details
Main Author: Wang, Minghao
Other Authors: Lihui Chen
Format: Thesis-Master by Coursework
Language:English
Published: Nanyang Technological University 2024
Subjects:
Engineering
Sentence embeddings
Online Access:https://hdl.handle.net/10356/173839
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Nanyang Technological University
Language: English
id sg-ntu-dr.10356-173839
record_format dspace
spelling sg-ntu-dr.10356-1738392024-03-01T15:44:20Z Large language model enhanced with prompt-based vanilla distillation for sentence embeddings Wang, Minghao Lihui Chen School of Electrical and Electronic Engineering ELHCHEN@ntu.edu.sg Engineering Sentence embeddings In this dissertation, the prompt-based method PromptEOL is used to train the opt- 2.7b model with the Parameter-Efficient Fine-Tuning method to reduce the number of training parameters and GPU memory usage. Then the opt-2.7b-lora model is used as the teacher model to train the student model under the distillation framework of DistillCSE with the vanilla distillation. The core method of evaluation we use centers on Semantic Textual Similarity detection. Master's degree 2024-03-01T02:52:12Z 2024-03-01T02:52:12Z 2023 Thesis-Master by Coursework Wang, M. (2023). Large language model enhanced with prompt-based vanilla distillation for sentence embeddings. Master's thesis, Nanyang Technological University, Singapore. https://hdl.handle.net/10356/173839 https://hdl.handle.net/10356/173839 en application/pdf Nanyang Technological University
institution Nanyang Technological University
building NTU Library
continent Asia
country Singapore
Singapore
content_provider NTU Library
collection DR-NTU
language English
topic Engineering
Sentence embeddings
spellingShingle Engineering
Sentence embeddings
Wang, Minghao
Large language model enhanced with prompt-based vanilla distillation for sentence embeddings
description In this dissertation, the prompt-based method PromptEOL is used to train the opt- 2.7b model with the Parameter-Efficient Fine-Tuning method to reduce the number of training parameters and GPU memory usage. Then the opt-2.7b-lora model is used as the teacher model to train the student model under the distillation framework of DistillCSE with the vanilla distillation. The core method of evaluation we use centers on Semantic Textual Similarity detection.
author2 Lihui Chen
author_facet Lihui Chen
Wang, Minghao
format Thesis-Master by Coursework
author Wang, Minghao
author_sort Wang, Minghao
title Large language model enhanced with prompt-based vanilla distillation for sentence embeddings
title_short Large language model enhanced with prompt-based vanilla distillation for sentence embeddings
title_full Large language model enhanced with prompt-based vanilla distillation for sentence embeddings
title_fullStr Large language model enhanced with prompt-based vanilla distillation for sentence embeddings
title_full_unstemmed Large language model enhanced with prompt-based vanilla distillation for sentence embeddings
title_sort large language model enhanced with prompt-based vanilla distillation for sentence embeddings
publisher Nanyang Technological University
publishDate 2024
url https://hdl.handle.net/10356/173839
_version_ 1794549360750493696

Similar Items