DEVELOPMENT AND IMPLEMENTATION OF COMMUNICATION MODIFICATION ON ADAM OPTIMIZER FOR DISTRIBUTED DEEP LEARNING

• Main Author: Febryan Suryawan, Fransiskus
• Format: Final Project
• Language: Indonesia
• Online Access: https://digilib.itb.ac.id/gdl/view/74150
• Institution: Institut Teknologi Bandung

Deep learning is a technique used in many domains to solve various problems. One of the popular example at the time of writing is ChatGPT, which uses GPT model made by OpenAI. A deep learning model consists of several layers, each with its own parameters. More complex models are developed by increasing the number of learned parameters. Increase of parameters in the model will cause communication overhead when learning in distributed architecture. Therefore, this final project will study about existing techniques to reduce communication overhead using reduction in commu- nication rounds and compression. This final project will review CADA (T. Chen dkk., 2021) and Efficient-Adam (C. Chen dkk., 2022). Then, this final project will incorporate ideas from both techniques, resulting in a technique which could reduce communications to as much as 0.97 times the communication rounds of CADA and using only 0.29 times the communication size of CADA.