Knowledge distillation in computer vision models

• Main Author: Yeoh, Yu Shyan
• Other Authors: Lin Guosheng
• Format: Final Year Project
• Language: English
• Published: Nanyang Technological University 2024
• Subjects: Computer and Information Science; Classification
• Online Access: https://hdl.handle.net/10356/181128
• Institution: Nanyang Technological University

Knowledge distillation has gained significant popularity in the Vision Transformer (ViT) space as a powerful approach to enhance the efficiency of a small lightweight model. Knowledge distillation enables a larger and complex “teacher” model to relay its knowledge to a smaller “student” model. This enables the student model to improve its own accuracy and retain its computational efficiency. Recent works, however, lack comprehensive exploration for Hybrid distillation techniques. This includes combining various distillation strategies to boost the efficiency of the student model. This project aims to research Hybrid distillation in the context of ViT models for image classification tasks. A series of experiments were conducted to compare the result of fine-tuned teacher and student models with distilled student models, including both traditional and Hybrid distillation approaches. The experiments on Hybrid distillation have shown to improve the accuracy of smaller student models with minimal impact on inference time, providing a possible solution for real-world applications.