Kun Yuan, Assistant Professor, Peking University

Bin Gao, Associate Professor

Accelerating Decentralized Deep Training with Sparse and Effective Topologies

16:00-17:00 November 9 (Thursday) , Lecture hall on the first floor of Siyuan Building

Decentralized optimization algorithms save remarkable communication overheads in distributed deep learning since each node averages locally with neighbors. The network topology connecting all nodes determines communication efficiency and local averaging effectiveness. The key to making decentralized algorithms efficient is to seek sparse topologies that realize effective local averaging with little communication. However, exiting common topologies either suffer expensive per-iteration communication or slow consensus rates. In this talk, we will propose several sparse and effective topologies that endow decentralized algorithms with state-of-the-art balance between communication efficiency and convergence. We will also discuss BlueFog, an open-source python library for straightforward, high-performance implementations of diverse topologies and decentralized algorithms.

Bio: Dr. Kun Yuan is an Assistant Professor at Center for Machine Learning Research (CMLR) in Peking University. He completed his Ph.D. degree at UCLA in 2019, and was a staff algorithm engineer in Alibaba (US) Group between 2019 and 2022. His research focuses on the development of fast, scalable, reliable, and distributed algorithms with applications in large-scale optimization, deep neural network training, federated learning, and Internet of Things. He was the recipient of the 2017 IEEE Signal Processing Society Young Author Best Paper Award, and the 2017 ICCM Distinguished Paper Award.