A Practical Human Labeling Method for Online Just-in-Time Software Defect Prediction

Song, Liyan1,2; Minku, Leandro3; Teng, Cong1,2; Yao, Xin1,2

10.1145/3611643.3616307

2023

31st ACM Joint Meeting of the European Software Engineering Conference / Symposium on the Foundations-of-Software-Engineering (ESEC/FSE)

PROCEEDINGS OF THE 31ST ACM JOINT MEETING EUROPEAN SOFTWARE ENGINEERING CONFERENCE AND SYMPOSIUM ON THE FOUNDATIONS OF SOFTWARE ENGINEERING, ESEC/FSE 2023

DEC 03-09, 2023

null,San Francisco,CA

1601 Broadway, 10th Floor, NEW YORK, NY, UNITED STATES

ASSOC COMPUTING MACHINERY

Just-in-Time Software Defect Prediction (JIT-SDP) can be seen as an online learning problem where additional software changes produced over time may be labeled and used to create training examples. These training examples form a data stream that can be used to update JIT-SDP models in an attempt to avoid models becoming obsolete and poorly performing. However, labeling procedures adopted in existing online JIT-SDP studies implicitly assume that practitioners would not inspect software changes upon a defect-inducing prediction, delaying the production of training examples. This is inconsistent with a real-world scenario where practitioners would adopt JIT-SDP models and inspect certain software changes predicted as defect-inducing to check whether they really induce defects. Such inspection means that some software changes would be labeled much earlier than assumed in existing work, potentially leading to different JIT-SDP models and performance results. This paper aims at formulating a more practical human labeling procedure that takes into account the adoption of JIT-SDP models during the software development process. It then analyses whether and to what extent it would impact the predictive performance of JIT-SDP models. We also propose a new method to target the labeling of software changes with the aim of saving human inspection effort. Experiments based on 14 GitHub projects revealed that adopting a more realistic labeling procedure led to significantly higher predictive performance than when delaying the labeling process, meaning that existing work may have been underestimating the performance of JIT-SDP. In addition, our proposed method to target the labeling process was able to reduce human effort while maintaining predictive performance by recommending practitioners to inspect software changes that are more likely to induce defects. We encourage the adoption of more realistic human labeling methods in research studies to obtain an evaluation of JIT-SDP predictive performance that is closer to reality.

Just-in-time software defect prediction online learning verification latency waiting time human labeling human inspection

第一 ; 通讯

英语

EI ; CPCI-S

National Natural Science Foundation of China (NSFC)["62002148","62250710682"]

Computer Science

Computer Science, Software Engineering ; Computer Science, Theory & Methods

WOS:001148157800050

Web of Science

1.Research Institute of Trustworthy Autonomous Systems, Southern University of Science and Technology, Shenzhen, China
2.Guangdong Provincial Key Laboratory of Braininspired Intelligent Computation, Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China
3.School of Computer Science, The University of Birmingham, Birmingham, Edgbaston, United Kingdom

Song, Liyan,Minku, Leandro,Teng, Cong,et al. A Practical Human Labeling Method for Online Just-in-Time Software Defect Prediction[C]. 1601 Broadway, 10th Floor, NEW YORK, NY, UNITED STATES:ASSOC COMPUTING MACHINERY,2023.