A Procedure to Continuously Evaluate Predictive Performance of Just-In-Time Software Defect Prediction Models During Software Development

Song，Liyan1; Minku，Leandro L.2

2022

10.1109/TSE.2022.3158831

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING   影响因子和分区

0098-5589

1939-3520

Just-In-Time Software Defect Prediction (JIT-SDP) uses machine learning to predict whether software changes are defect-inducing or clean. When adopting JIT-SDP, changes in the underlying defect generating process may significantly affect the predictive performance of JIT-SDP models over time. Therefore, being able to continuously track the predictive performance of JIT-SDP models during the software development process is of utmost importance for software companies to decide whether or not to trust the predictions provided by such models over time. However, there has been little discussion on how to continuously evaluate predictive performance in practice, and such evaluation is not straightforward. In particular, labeled software changes that can be used for evaluation arrive over time with a delay, which in part corresponds to the time we have to wait to label software changes as ‘clean’ (waiting time). A clean label assigned based on a given waiting time may not correspond to the true label of the software changes. This can potentially hinder the validity of any continuous predictive performance evaluation procedure for JIT-SDP models. This paper provides the first discussion of how to continuously evaluate predictive performance of JIT-SDP models over time during the software development process, and the first investigation of whether and to what extent waiting time affects the validity of such continuous performance evaluation procedure in JIT-SDP. Based on 13 GitHub projects, we found that waiting time had a significant impact on the validity. Though typically small, the differences in estimated predicted performance were sometimes large, and thus inappropriate choices of waiting time can lead to misleading estimations of predictive performance over time. Such impact did not normally change the ranking between JIT-SDP models, and thus conclusions in terms of which JIT-SDP model performs better are likely reliable independent of the choice of waiting time, especially when considered across projects.

Delays Estimation Performance evaluation Predictive models Software Software reliability Training

EI ; SCI

英语

第一

National Natural Science Foundation of China[62002148] ; EPSRC[EP/R006660/2] ; Guangdong Provincial Key Laboratory[2020B121201001] ; Program for Guangdong Introducing Innovative and Enterpreneurial Teams[2017ZT07X386] ; Shenzhen Science and Technology Program[KQTD2016112514355531]

Computer Science ; Engineering

Computer Science, Software Engineering ; Engineering, Electrical & Electronic

WOS:000937151900010

IEEE COMPUTER SOC

20221211818319

Defects ; Forecasting ; Learning systems ; Software design ; Software reliability

Computer Programming:723.1 ; Computer Applications:723.5 ; Materials Science:951

COMPUTER SCIENCE

2-s2.0-85126517211

Scopus

1.Department of Computer Science and Engineering, Southern University of Science and Technology, 255310 Shenzhen, Guangdong, China, 518055
2.School of Computer Science, University of Birmingham, 1724 Birmingham, West Midlands, United Kingdom of Great Britain and Northern Ireland, B15 2TT

Song，Liyan,Minku，Leandro L.. A Procedure to Continuously Evaluate Predictive Performance of Just-In-Time Software Defect Prediction Models During Software Development[J]. IEEE TRANSACTIONS ON SOFTWARE ENGINEERING,2022,PP(99):1-1.

Song，Liyan,&Minku，Leandro L..(2022).A Procedure to Continuously Evaluate Predictive Performance of Just-In-Time Software Defect Prediction Models During Software Development.IEEE TRANSACTIONS ON SOFTWARE ENGINEERING,PP(99),1-1.

Song，Liyan,et al."A Procedure to Continuously Evaluate Predictive Performance of Just-In-Time Software Defect Prediction Models During Software Development".IEEE TRANSACTIONS ON SOFTWARE ENGINEERING PP.99(2022):1-1.