Win: Weight-decay-integrated Nesterov acceleration for faster network training

• Main Authors: ZHOU, Pan, XIE, Xingyu, LIN, Zhouchen, TOH, Kim-Chuan, YAN, Shuicheng
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2024
• Subjects: Accelerated Adaptive Gradient Algorithms; Deep Learning Optimizer; Network Optimization; Nesterov Acceleration in Deep Learning; OS and Networks; Theory and Algorithms
• Online Access: https://ink.library.smu.edu.sg/sis_research/8969; https://ink.library.smu.edu.sg/context/sis_research/article/9972/viewcontent/2024JMLR.pdf
• Institution: Singapore Management University

Training deep networks on large-scale datasets is computationally challenging. This work explores the problem of “how to accelerate adaptive gradient algorithms in a general manner", and proposes an effective Weight-decay-Integrated Nesterov acceleration (Win) to accelerate adaptive algorithms....