Win: Weight-decay-integrated nesterov acceleration for adaptive gradient algorithms

• Main Authors: ZHOU, Pan, XIE, Xingyu, YAN, Shuicheng
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2023
• Subjects: Network optimizers; Deep learning optimizer; Deep learning algorithm; Optimization acceleration in deep learning; Deep Learning and representational learning; OS and Networks; Theory and Algorithms
• Online Access: https://ink.library.smu.edu.sg/sis_research/9056; https://ink.library.smu.edu.sg/context/sis_research/article/10059/viewcontent/653_win_weight_decay_integrated_ICLR.pdf
• Institution: Singapore Management University

Training deep networks on large-scale datasets is computationally challenging. In this work, we explore the problem of “how to accelerate adaptive gradient algorithms in a general manner”, and aim to provide practical efficiency-boosting insights. To this end, we propose an effective and general Wei...