Deeproad: GaN-based metamorphic testing and input validation framework for autonomous driving systems

Zhang，Mengshi1; Zhang，Yuqun2; Zhang，Lingming3; Liu，Cong3; Khurshid，Sarfraz1

10.1145/3238147.3238187

2018-09-03

1527-1366

978-1-7281-4376-7

ASE 2018 - Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering

132-142

3-7 Sept. 2018

Montpellier, France

345 E 47TH ST, NEW YORK, NY 10017 USA

Association for Computing Machinery, Inc

While Deep Neural Networks (DNNs) have established the fundamentals of image-based autonomous driving systems, they may exhibit erroneous behaviors and cause fatal accidents. To address the safety issues in autonomous driving systems, a recent set of testing techniques have been designed to automatically generate artificial driving scenes to enrich test suite, e.g., generating new input images transformed from the original ones. However, these techniques are insufficient due to two limitations: first, many such synthetic images often lack diversity of driving scenes, and hence compromise the resulting efficacy and reliability. Second, for machine-learning-based systems, a mismatch between training and application domain can dramatically degrade system accuracy, such that it is necessary to validate inputs for improving system robustness. In this paper, we propose DeepRoad, an unsupervised DNN-based framework for automatically testing the consistency of DNN-based autonomous driving systems and online validation. First, DeepRoad automatically synthesizes large amounts of diverse driving scenes without using image transformation rules (e.g. scale, shear and rotation). In particular, DeepRoad is able to produce driving scenes with various weather conditions (including those with rather extreme conditions) by applying Generative Adversarial Networks (GANs) along with the corresponding real-world weather scenes. Second, DeepRoad utilizes metamorphic testing techniques to check the consistency of such systems using synthetic images. Third, DeepRoad validates input images for DNN-based systems by measuring the distance of the input and training images using their VGGNet features. We implement DeepRoad to test three well-recognized DNN-based autonomous driving systems in Udacity self-driving car challenge. The experimental results demonstrate that DeepRoad can detect thousands of inconsistent behaviors for these systems, and effectively validate input images to potentially enhance the system robustness as well.

Deep neural networks Input validation Software testing Test generation

通讯

英语

EI ; CPCI-S

Ministry of Science and Technology of the People's Republic of China[2017YFC0804002]

Computer Science

Computer Science, Software Engineering ; Computer Science, Theory & Methods

WOS:000553784500016

20184706087442

Automatic test pattern generation ; Deep neural networks ; Gallium nitride ; III-V semiconductors ; Online systems ; Safety testing ; Software testing

Digital Computers and Systems:722.4 ; Computer Applications:723.5 ; Accidents and Accident Prevention:914.1

2-s2.0-85056509092

Scopus

1.University of Texas, ,Austin,United States
2.Shenzhen Key Laboratory of Computational Intelligence, Department of Computer Science and Engineering, Southern University of Science and Technology, ,China
3.University of Texas, ,Dallas,United States

Zhang，Mengshi,Zhang，Yuqun,Zhang，Lingming,et al. Deeproad: GaN-based metamorphic testing and input validation framework for autonomous driving systems[C]. 345 E 47TH ST, NEW YORK, NY 10017 USA:Association for Computing Machinery, Inc,2018:132-142.