Minimizing Network Traffic for Distributed Joins Using Lightweight Locality-Aware Scheduling

Cheng, Long1,2; Murphy, John1; Liu, Qingzhi2; Hao, Chunliang3; Theodoropoulos, Georgios4

10.1007/978-3-319-96983-1_21

2018

16113349

24th International European Conference on Parallel and Distributed Computing, Euro-Par 2018

11014 LNCS

293-305

Turin, Italy

Springer Verlag

Large computing systems such as data centers are becoming the mainstream infrastructures for big data processing. As one of the key data operators in such scenarios, distributed joins is still challenging current techniques since it always incurs a significant cost on network communication. Various advanced approaches have been proposed to improve the performance, however, most of them just focus on data skew handling, and algorithms designed specifically for communication reduction have received less attention. Moreover, although the state-of-the-art technique can minimize network traffic, it provides fine-grained optimal schedules for all individual join keys, which could result in obvious overhead. In this paper, we propose a new approach called LAS (Lightweight Locality-Aware Scheduling), which targets reducing network communication for large distributed joins in an efficient and effective manner. We present the detailed design and implementation of LAS, and conduct an experimental evaluation using large data joins. Our results show that LAS can effectively reduce scheduling overhead and achieve comparable performance on network reduction compared to the state-of-the-art.
© 2018, Springer International Publishing AG, part of Springer Nature.

其他

EI

[799066]

20183705797336

Computer networks ; Data handling ; Distributed computer systems ; Scheduling

Digital Computers and Systems:722.4 ; Data Processing and Image Processing:723.2 ; Management:912.2

EV Compendex

1.PEL, University College Dublin, Dublin, Ireland
2.Eindhoven University of Technology, Eindhoven, Netherlands
3.Institute of Software, CAS, Beijing, China
4.Southern University of Science and Technology, Shenzhen, China

Cheng, Long,Murphy, John,Liu, Qingzhi,et al. Minimizing Network Traffic for Distributed Joins Using Lightweight Locality-Aware Scheduling[C]:Springer Verlag,2018:293-305.