3PC: Three point compressors for communication-efficient distributed training and a better theory for lazy aggregation

• Main Authors: RICHTARIK, Peter, SOKOLOV, Igor, FATKHULLIN, Ilyas, GASANOV, Elnur, LI, Zhize, GORBUNOV, Eduard
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2022
• Subjects: Databases and Information Systems
• Online Access: https://ink.library.smu.edu.sg/sis_research/8685; https://ink.library.smu.edu.sg/context/sis_research/article/9688/viewcontent/ICML22_full_3pc.pdf
• Institution: Singapore Management University

We propose and study a new class of gradient communication mechanisms for communication-efficient training -- three point compressors (3PC) -- as well as efficient distributed nonconvex optimization algorithms that can take advantage of them. Unlike most established approaches, which rely on a stati...