EvalDNN: A toolbox for evaluating deep neural network models

Tian，Yongqiang1; Zeng，Zhihua2; Wen，Ming3; Liu，Yepang4; Kuo，Tzu Yang1; Cheung，Shing Chi1

10.1145/3377812.3382133

2020-06-27

0270-5257

978-1-7281-6528-8

Proceedings - International Conference on Software Engineering

45-48

5-11 Oct. 2020

Seoul, Korea (South)

Recent studies have shown that the performance of deep learningmodels should be evaluated using various important metrics suchas robustness and neuron coverage, besides the widely-used prediction accuracy metric. However, major deep learning frameworkscurrently only provide APIs to evaluate a model's accuracy. In order to comprehensively assess a deep learning model, frameworkusers and researchers often need to implement new metrics bythemselves, which is a tedious job. What is worse, due to the largenumber of hyper-parameters and inadequate documentation, evaluation results of some deep learning models are hard to reproduce,especially when the models and metrics are both new.To ease the model evaluation in deep learning systems, we havedeveloped EvalDNN, a user-friendly and extensible toolbox supporting multiple frameworks and metrics with a set of carefullydesigned APIs. Using EvalDNN, evaluation of a pre-trained modelwith respect to different metrics can be done with a few lines ofcode. We have evaluated EvalDNN on 79 models from TensorFlow,Keras, GluonCV, and PyTorch. As a result of our effort made toreproduce the evaluation results of existing work, we release aperformance benchmark of popular models, which can be a useful reference to facilitate future research. The tool and benchmarkare available at https://github.com/yqtianust/EvalDNN and https://yqtianust.github.io/EvalDNN-benchmark/, respectively. A demovideo of EvalDNN is available at: https://youtu.be/v69bNJN2bJc.

Deep Learning Model Evaluation

其他

英语

EI

20204409420993

HTTP ; Neural network models ; Deep neural networks

Ergonomics and Human Factors Engineering:461.4 ; Artificial Intelligence:723.4

2-s2.0-85094112889

Scopus

1.Hong Kong University of Science and Technology,Hong Kong,Hong Kong
2.Zhejiang University,Hangzhou,China
3.Huazhong University of Science and Technology,Wuhan,China
4.Southern University of Science and Technology,Shenzhen,China

Tian，Yongqiang,Zeng，Zhihua,Wen，Ming,et al. EvalDNN: A toolbox for evaluating deep neural network models[C],2020:45-48.