DHive: Query Execution Performance Analysis via Dataflow in Apache Hive

Zhang, Chaozu1; Shen, Qiaomu2; Tang, Bo1

2023-08-01

10.14778/3611540.3611605

PROCEEDINGS OF THE VLDB ENDOWMENT   影响因子和分区

2150-8097

Nowadays, Apache Hive has been widely used for large-scale data analysis applications in many organizations. Various visual analytical tools are developed to help Hive users quickly analyze the query execution process and identify the performance bottleneck of executed queries. However, existing tools mostly focus on showing the time usage of query sub-components (jobs and operators) but fail to provide enough evidence to analyze the root reasons for the slow execution progress. To tackle this problem, we develop a visual analytical system DHive to visualize and analyze the query execution progress via dataflow analysis. DHive shows the dataflow during query execution at multiple levels: query level, job level and task level, which enable users to identify the key jobs/tasks and explain their time usage by linking them to the auxiliary information such as the system configuration and hardware status. We demonstrate the effectiveness of DHive by two cases in a production cluster. DHive is open-source at https://github.com/DBGroupSUSTech/DHive.git.

SCI ; EI

英语

第一 ; 通讯

Shenzhen Fundamental Research Program[20220815112848002] ; Guangdong Provincial Key Laboratory[2020B121201001]

Computer Science

Computer Science, Information Systems ; Computer Science, Theory & Methods

WOS:001067701000066

ASSOC COMPUTING MACHINERY

20234314943565

Data flow analysis

Computer Software, Data Handling and Applications:723

Web of Science

1.Southern Univ Sci & Technol, Dept Comp Sci & Engn, Shenzhen, Peoples R China
2.Southern Univ Sci & Technol, Res Inst Trustworthy Autonomous Syst, Shenzhen, Peoples R China

Zhang, Chaozu,Shen, Qiaomu,Tang, Bo. DHive: Query Execution Performance Analysis via Dataflow in Apache Hive[J]. PROCEEDINGS OF THE VLDB ENDOWMENT,2023,16(12):3998-4001.

Zhang, Chaozu,Shen, Qiaomu,&Tang, Bo.(2023).DHive: Query Execution Performance Analysis via Dataflow in Apache Hive.PROCEEDINGS OF THE VLDB ENDOWMENT,16(12),3998-4001.

Zhang, Chaozu,et al."DHive: Query Execution Performance Analysis via Dataflow in Apache Hive".PROCEEDINGS OF THE VLDB ENDOWMENT 16.12(2023):3998-4001.