Improving the robustness and performance of parallel joins over distributed systems

Cheng, Long1,2; Kotoulas, Spyros3; Ward, Tomas E.4; Theodoropoulos, Georgios5

2017-11

10.1016/j.jpdc.2017.06.016

JOURNAL OF PARALLEL AND DISTRIBUTED COMPUTING   影响因子和分区

0743-7315

1096-0848

High-performance data processing systems typically utilize numerous servers with large amounts of memory. An essential operation in such environment is the parallel join, the performance of which is critical for data intensive operations. In many real-world workloads, data skew is omnipresent. Techniques that do not cater for the possibility of data skew often suffer from performance failures and memory problems. State-of-the-art methods designed to handle data skew propose new ways to distribute computation that avoid hotspots. However, this comes at the expense of global collection of statistics, redundant computation, duplication of data or increased network communication. In this light, performance could be further improved by removing the dependency on global skew knowledge and broadcasting. In this paper, we propose a new method called PRPQ (partial redistribution & partial query), with targets for efficient and robust joins with large datasets over high performance clusters. We present the detailed implementation of our approach and compare its performance with current implementations. The experimental results demonstrate that the proposed algorithm is scalable and robust and can also outperform the state-of-the-art approach with less network communication, figures that confirm our theoretical analysis. (C) 2017 Elsevier Inc. All rights reserved.

Parallel joins Data skew Robust High performance computing

SCI ; EI

英语

其他

Emmy Noether grant[KR 4381/1-1]

Computer Science

Computer Science, Theory & Methods

WOS:000408298400023

ACADEMIC PRESS INC ELSEVIER SCIENCE

20173003966884

Artificial intelligence ; Computer programming

Computer Software, Data Handling and Applications:723

COMPUTER SCIENCE

Web of Science

1.Eindhoven Univ Technol, Eindhoven, Netherlands
2.Tech Univ Dresden, Dresden, Germany
3.IBM Res, Dublin, Ireland
4.Maynooth Univ, Maynooth, Kildare, Ireland
5.Southern Univ Sci & Technol, Shenzhen, Peoples R China

Cheng, Long,Kotoulas, Spyros,Ward, Tomas E.,et al. Improving the robustness and performance of parallel joins over distributed systems[J]. JOURNAL OF PARALLEL AND DISTRIBUTED COMPUTING,2017,109:310-323.

Cheng, Long,Kotoulas, Spyros,Ward, Tomas E.,&Theodoropoulos, Georgios.(2017).Improving the robustness and performance of parallel joins over distributed systems.JOURNAL OF PARALLEL AND DISTRIBUTED COMPUTING,109,310-323.

Cheng, Long,et al."Improving the robustness and performance of parallel joins over distributed systems".JOURNAL OF PARALLEL AND DISTRIBUTED COMPUTING 109(2017):310-323.