Hardware-constrained edge deep learning
Neural Networks have become commonplace in our daily lives, powering everything from language models in chatbots to computer vision models in industrial machinery. The unending quest for greater model performance has led to an exponential growth in model size. For many devices, especially edge dev...
Saved in:
| 主要作者: | |
|---|---|
| 其他作者: | |
| 格式: | Final Year Project |
| 語言: | English |
| 出版: |
Nanyang Technological University
2024
|
| 主題: | |
| 在線閱讀: | https://hdl.handle.net/10356/181190 |
| 標簽: |
添加標簽
沒有標簽, 成為第一個標記此記錄!
|
| 機構: | Nanyang Technological University |
| 語言: | English |
| 總結: | Neural Networks have become commonplace in our daily lives, powering everything from language models in chatbots to computer vision models in industrial machinery.
The unending quest for greater model performance has led to an exponential growth in model size.
For many devices, especially edge devices, storing or even running these models in a performant manner proves to be a challenge.
In this paper, various memory compression methods, centered around post-training quantization, are explored for Large Language Models (LLMs) by comparing accuracy (perplexity) and inference latency (token generation speed).
The report concludes that most LLMs can be quantized significantly without an observable loss in accuracy.
However, very aggressive quantization (\<=3 bits) can lead to rambling responses and a significant degradation in user experience.
Further work can also be done to explore kernel-level quantization for convolutional neural networks and pseudo-vectorization for embedded use cases. |
|---|