A systematic evaluation of problematic tests generated by evosuite

Fan, Zhiyu

10.1109/ICSE-Companion.2019.00068

2019

2574-1926

978-1-7281-1765-2

41st IEEE/ACM International Conference on Software Engineering: Companion, ICSE-Companion 2019

165-167

25-31 May 2019

Montreal, QC, Canada

345 E 47TH ST, NEW YORK, NY 10017 USA

Institute of Electrical and Electronics Engineers Inc.

With the rapidly growing scale of modern software, the reliability of software systems has become essential. To ease the developers' pressure of writing unit tests manually, test generation tools such as EvoSuite and Randoop were proposed. Although these approaches have been shown to be able to automatically generate tests for achieving high coverage, the generated tests may be ineffective in detecting real faults. Particularly, these automatically generated tests may suffer from several problems (we call them problematic tests): (1) incorrect oracle. (2) unexpected exception/error. (3) flaky test. We present a comprehensive study of EvoSuite in Defects4j, and performed a detailed analysis of the reasons behind these automatically generated problematic tests. Our analysis identifies 528 problematic tests: 208 (39.4%) of them are caused by incorrect oracle, 319 (60.4%) are caused by unexpected exception/error, and one flaky test.
© 2019 IEEE.

Automated Test generation Empirical Study Flaky tests

第一 ; 通讯

英语

EI ; CPCI-S

Computer Science

Computer Science, Software Engineering

WOS:000503272600049

20193707414539

Software reliability

Computer Applications:723.5 ; Automatic Control Principles and Applications:731

EV Compendex

Southern University of Science and Technology, China

Fan, Zhiyu. A systematic evaluation of problematic tests generated by evosuite[C]. 345 E 47TH ST, NEW YORK, NY 10017 USA:Institute of Electrical and Electronics Engineers Inc.,2019:165-167.