Protocol Customization for Improving MPI Performance on RDMA-Enabled Clusters

Zheng Gu, Matthew Small, Xin Yuan, Aniruddha Marathe, David K. Lowenthal

Research output: Contribution to journalArticlepeer-review

2 Scopus citations


Optimizing Message Passing Interface (MPI) point-to-point communication for large messages is of paramount importance since most communications in MPI applications are performed by such operations. Remote Direct Memory Access (RDMA) allows one-sided data transfer and provides great flexibility in the design of efficient communication protocols for large messages. However, achieving high point-to-point communication performance on RDMA-enabled clusters is challenging due to both the complexity in communication protocols and the impact of the protocol invocation scenario on the performance of a given protocol. In this work, we analyze existing protocols and show that they are not ideal in many situations, and propose to use protocol customization, that is, different protocols for different situations to improve MPI performance. More specifically, by leveraging the RDMA capability, we develop a set of protocols that can provide high performance for all protocol invocation scenarios. Armed with this set of protocols that can collectively achieve high performance in all situations, we demonstrate the potential of protocol customization by developing a trace-driven toolkit that allows the appropriate protocol to be selected for each communication in an MPI application to maximize performance. We evaluate the performance of the proposed techniques using micro-benchmarks and application benchmarks. The results indicate that protocol customization can out-perform traditional communication schemes by a large degree in many situations.

Original languageEnglish (US)
Pages (from-to)682-703
Number of pages22
JournalInternational Journal of Parallel Programming
Issue number5
StatePublished - Oct 2013
Externally publishedYes


  • MPI
  • Point-to-point communication
  • Protocol customization

ASJC Scopus subject areas

  • Software
  • Theoretical Computer Science
  • Information Systems


Dive into the research topics of 'Protocol Customization for Improving MPI Performance on RDMA-Enabled Clusters'. Together they form a unique fingerprint.

Cite this