Towards understanding convergence and generalization of AdamW

• المؤلفون الرئيسيون: ZHOU, Pan, XIE, Xingyu, LIN, Zhouchen, YAN, Shuicheng
• التنسيق: text
• اللغة: English
• منشور في: Institutional Knowledge at Singapore Management University 2024
• الموضوعات: Analysis of AdamW; Convergence of AdamW; Generalization of AdamW; Adaptive gradient algorithms; Graphics and Human Computer Interfaces
• الوصول للمادة أونلاين: https://ink.library.smu.edu.sg/sis_research/8986; https://ink.library.smu.edu.sg/context/sis_research/article/9989/viewcontent/2023_TPAMI_AdamW_Analysis.pdf

AdamW modifies Adam by adding a decoupled weight decay to decay network weights per training iteration. For adaptive algorithms, this decoupled weight decay does not affect specific optimization steps, and differs from the widely used ℓ2-regularizer which changes optimization steps via changing the...