On the worst-case communication overhead for distributed data shuffling

Mohamed Adel Attia, Ravi Tandon

Research output: Chapter in Book/Report/Conference proceedingConference contribution

10 Scopus citations

Abstract

Distributed learning platforms for processing large scale data-sets are becoming increasingly prevalent. In typical distributed implementations, a centralized master node breaks the data-set into smaller batches for parallel processing across distributed workers to achieve speed-up and efficiency. Several computational tasks are of sequential nature, and involve multiple passes over the data. At each iteration over the data, it is common practice to randomly re-shuffle the data at the master node, assigning different batches for each worker to process. This random re-shuffling operation comes at the cost of extra communication overhead, since at each shuffle, new data points need to be delivered to the distributed workers. In this paper, we focus on characterizing the information theoretically optimal communication overhead for the distributed data shuffling problem. We propose a novel coded data delivery scheme for the case of no excess storage, where every worker can only store the assigned data batches under processing. Our scheme exploits a new type of coding opportunity and is applicable to any arbitrary shuffle, and for any number of workers. We also present information theoretic lower bounds on the minimum communication overhead for data shuffling, and show that the proposed scheme matches this lower bound for the worst-case communication overhead.

Original languageEnglish (US)
Title of host publication54th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2016
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages961-968
Number of pages8
ISBN (Electronic)9781509045495
DOIs
StatePublished - Feb 10 2017
Event54th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2016 - Monticello, United States
Duration: Sep 27 2016Sep 30 2016

Publication series

Name54th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2016

Conference

Conference54th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2016
Country/TerritoryUnited States
CityMonticello
Period9/27/169/30/16

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computational Theory and Mathematics
  • Computer Networks and Communications
  • Hardware and Architecture
  • Control and Optimization

Fingerprint

Dive into the research topics of 'On the worst-case communication overhead for distributed data shuffling'. Together they form a unique fingerprint.

Cite this