Hardware-constrained edge deep learning

• Main Author: Ng, Jia Rui
• Other Authors: Weichen Liu
• Format: Final Year Project
• Language: English
• Published: Nanyang Technological University 2024
• Subjects: Computer and Information Science
• Online Access: https://hdl.handle.net/10356/181190
• Institution: Nanyang Technological University

Neural Networks have become commonplace in our daily lives, powering everything from language models in chatbots to computer vision models in industrial machinery. The unending quest for greater model performance has led to an exponential growth in model size. For many devices, especially edge devices, storing or even running these models in a performant manner proves to be a challenge. In this paper, various memory compression methods, centered around post-training quantization, are explored for Large Language Models (LLMs) by comparing accuracy (perplexity) and inference latency (token generation speed). The report concludes that most LLMs can be quantized significantly without an observable loss in accuracy. However, very aggressive quantization (\<=3 bits) can lead to rambling responses and a significant degradation in user experience. Further work can also be done to explore kernel-level quantization for convolutional neural networks and pseudo-vectorization for embedded use cases.