Discriminator-enhanced knowledge-distillation networks

• Main Authors: Li, Zhenping, Cao, Zhen, Li, Pengfei, Zhong, Yong, Li, Shaobo
• Other Authors: School of Electrical and Electronic Engineering
• Format: Article
• Language: English
• Published: 2023
• Subjects: Engineering::Electrical and electronic engineering; Knowledge Distillation; Reinforcement Learning
• Online Access: https://hdl.handle.net/10356/171765
• Institution: Nanyang Technological University

Query auto-completion (QAC) serves as a critical functionality in contemporary textual search systems by generating real-time query completion suggestions based on a user’s input prefix. Despite the prevalent use of language models (LMs) in QAC candidate generation, LM-based approaches frequently suffer from overcorrection issues during pair-wise loss training and efficiency deficiencies. To address these challenges, this paper presents a novel framework—discriminator-enhanced knowledge distillation (Dis-KD)—for the QAC task. This framework combines three core components: a large-scale pre-trained teacher model, a lightweight student model, and a discriminator for adversarial learning. Specifically, the discriminator aids in discerning generative-level differences between the teacher and the student models. An additional discriminator score loss is amalgamated with the traditional knowledge-distillation loss, resulting in enhanced performance of the student model. Contrary to the stepwise evaluation of each generated word, our approach assesses the entire generation sequence. This method alleviates the prevalent overcorrection issue in the generation process. Consequently, our proposed framework boasts improvements in model accuracy and a reduction in parameter size. Empirical results highlight the superiority of Dis-KD over established baseline methods, with the student model surpassing the teacher model in QAC tasks for sub-word languages.