Hardware-constrained edge deep learning
Neural Networks have become commonplace in our daily lives, powering everything from language models in chatbots to computer vision models in industrial machinery. The unending quest for greater model performance has led to an exponential growth in model size. For many devices, especially edge dev...
Saved in:
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Final Year Project |
| Language: | English |
| Published: |
Nanyang Technological University
2024
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/181190 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Institution: | Nanyang Technological University |
| Language: | English |
| Summary: | Neural Networks have become commonplace in our daily lives, powering everything from language models in chatbots to computer vision models in industrial machinery.
The unending quest for greater model performance has led to an exponential growth in model size.
For many devices, especially edge devices, storing or even running these models in a performant manner proves to be a challenge.
In this paper, various memory compression methods, centered around post-training quantization, are explored for Large Language Models (LLMs) by comparing accuracy (perplexity) and inference latency (token generation speed).
The report concludes that most LLMs can be quantized significantly without an observable loss in accuracy.
However, very aggressive quantization (\<=3 bits) can lead to rambling responses and a significant degradation in user experience.
Further work can also be done to explore kernel-level quantization for convolutional neural networks and pseudo-vectorization for embedded use cases. |
|---|