PROJECT TITLE :

Exploiting Efficient and Scalable Shuffle Transfers in Future Data Center Networks

ABSTRACT:

Distributed computing systems like MapReduce in knowledge centers transfer huge amount of data across successive processing stages. Such shuffle transfers contribute most of the network traffic and make the network bandwidth become a bottleneck. In many commonly used workloads, knowledge flows in such a transfer are highly correlated and aggregated at the receiver facet. To lower down the network traffic and efficiently use the on the market network bandwidth, we propose to push the aggregation computation into the network and parallelize the shuffle and reduce phases. During this paper, we initial examine the gain and feasibility of the in-network aggregation with BCube, a completely unique server-centric networking structure for future information centers. To exploit such a gain, we model the in-network aggregation drawback that's NP-onerous in BCube. We have a tendency to propose 2 approximate strategies for building the economical IRS-primarily based incast aggregation tree and SRS-based mostly shuffle aggregation subgraph, solely primarily based on the labels of their members and the data center topology. We any style scalable forwarding schemes primarily based on Bloom filters to implement in-network aggregation over large concurrent shuffle transfers. Based mostly on a prototype and large-scale simulations, we demonstrate that our approaches will considerably decrease the amount of network traffic and save the info center resources. Our approaches for BCube can be custom-made to different server-centric network structures for future knowledge centers once minimal modifications.