Towards understanding convergence and generalization of AdamW

• Main Authors: ZHOU, Pan, XIE, Xingyu, LIN, Zhouchen, YAN, Shuicheng
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2024
• Subjects: Analysis of AdamW; Convergence of AdamW; Generalization of AdamW; Adaptive gradient algorithms; Graphics and Human Computer Interfaces
• Online Access: https://ink.library.smu.edu.sg/sis_research/8986; https://ink.library.smu.edu.sg/context/sis_research/article/9989/viewcontent/2023_TPAMI_AdamW_Analysis.pdf
• Institution: Singapore Management University

AdamW modifies Adam by adding a decoupled weight decay to decay network weights per training iteration. For adaptive algorithms, this decoupled weight decay does not affect specific optimization steps, and differs from the widely used ℓ2-regularizer which changes optimization steps via changing the...