南方科技大学知识苑(SUSTech KC): Communication-Efficient Distributed Learning via Lazily Aggregated Quantized Gradients

题名	Communication-Efficient Distributed Learning via Lazily Aggregated Quantized Gradients
作者	Sun, Jun 1; Chen, Tianyi 2; Giannakis, Georgios B.3; Yang, Zaiyue4
通讯作者	Sun, Jun
发表日期	2019
ISSN	1049-5258
会议录名称	ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 32 (NIPS 2019)
卷号	32
出版地	10010 NORTH TORREY PINES RD, LA JOLLA, CALIFORNIA 92037 USA
出版者	NEURAL INFORMATION PROCESSING SYSTEMS (NIPS)
摘要	The present paper develops a novel aggregated gradient approach for distributed machine learning that adaptively compresses the gradient communication. The key idea is to first quantize the computed gradients, and then skip less informative quantized gradient communications by reusing outdated gradients. Quantizing and skipping result in 'lazy' worker-server communications, which justifies the term Lazily Aggregated Quantized gradient that is henceforth abbreviated as LAQ. Our LAQ can provably attain the same linear convergence rate as the gradient descent in the strongly convex case, while effecting major savings in the communication overhead both in transmitted bits as well as in communication rounds. Empirically, experiments with real data corroborate a significant communication reduction compared to existing gradient- and stochastic gradient-based algorithms.
学校署名	其他
语种	英语
相关链接	[来源记录]
收录类别	CPCI-S ; EI
资助项目	Shenzhen Committee on Science and Innovations[GJHZ20180411143603361] ; Department of Science and Technology of Guangdong Province[2018A050506003] ; Natural Science Foundation of China[61873118] ; NSF[1500713][1711471]
WOS研究方向	Computer Science
WOS类目	Computer Science, Artificial Intelligence
WOS记录号	WOS:000534424303037
EI入藏号	20203609141360
EI主题词	Gradient methods
EI分类号	Control Systems:731.1 ; Numerical Methods:921.6 ; Systems Science:961
来源库	Web of Science
引用统计	被引频次[WOS]：0
成果类型	会议论文
条目标识符	http://sustech.caswiz.com/handle/2SGJ60CL/139966
专题	南方科技大学
作者单位	1.Zhejiang Univ, Hangzhou 310027, Peoples R China 2.Rensselaer Polytech Inst, Troy, NY 12180 USA 3.Univ Minnesota, Minneapolis, MN 55455 USA 4.Southern Univ Sci & Technol, Shenzhen 518055, Peoples R China
推荐引用方式 GB/T 7714	Sun, Jun,Chen, Tianyi,Giannakis, Georgios B.,et al. Communication-Efficient Distributed Learning via Lazily Aggregated Quantized Gradients[C]. 10010 NORTH TORREY PINES RD, LA JOLLA, CALIFORNIA 92037 USA:NEURAL INFORMATION PROCESSING SYSTEMS (NIPS),2019.