MAC Aware Quantization for Distributed Gradient Descent

Wei Ting Chang, Ravi Tandon

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Scopus citations

Abstract

In this work, we study the problem of federated learning (FL), where distributed users aim to jointly train a machine learning model with the help of a parameter server (PS). In each iteration of FL, users compute local gradients, followed by transmission of the quantized gradients for subsequent aggregation and model updates at PS. One of the challenges of FL is that of communication overhead due to FL's iterative nature and large model sizes. One recent direction to alleviate communication bottleneck in FL is to let users communicate simultaneously over a multiple access channel (MAC), possibly making better use of the communication resources.In this paper, we consider the problem of FL over a MAC. We focus on the design of digital gradient transmission schemes over a MAC, where gradients at each user are first quantized, and then transmitted over a MAC to be decoded individually at the PS. When designing digital FL schemes over MACs, there are new opportunities to assign different amount of resources (e.g., rate or bandwidth) to different users based on a) the informativeness of the gradients at users, and b) the underlying channel conditions. We propose a stochastic gradient quantization scheme, where the quantization parameters are optimized based on the capacity region of the MAC. We show that such channel aware quantization} for FL outperforms uniform quantization, particularly when users experience different channel conditions, and when have gradients with varying levels of informativeness.

Original languageEnglish (US)
Title of host publication2020 IEEE Global Communications Conference, GLOBECOM 2020 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781728182988
DOIs
StatePublished - Dec 2020
Event2020 IEEE Global Communications Conference, GLOBECOM 2020 - Virtual, Taipei, Taiwan, Province of China
Duration: Dec 7 2020Dec 11 2020

Publication series

Name2020 IEEE Global Communications Conference, GLOBECOM 2020 - Proceedings

Conference

Conference2020 IEEE Global Communications Conference, GLOBECOM 2020
Country/TerritoryTaiwan, Province of China
CityVirtual, Taipei
Period12/7/2012/11/20

ASJC Scopus subject areas

  • Media Technology
  • Modeling and Simulation
  • Instrumentation
  • Artificial Intelligence
  • Computer Networks and Communications
  • Hardware and Architecture
  • Software
  • Safety, Risk, Reliability and Quality

Fingerprint

Dive into the research topics of 'MAC Aware Quantization for Distributed Gradient Descent'. Together they form a unique fingerprint.

Cite this