Optimization based layer-wise magnitude-based pruning for DNN compression

Li, Guiying1; Qian, Chao1; Jiang, Chunhui1; Lu, Xiaofen2; Tang, Ke3

2018

1045-0823

27th International Joint Conference on Artificial Intelligence, IJCAI 2018

2018-July

2383-2389

Stockholm, Sweden

International Joint Conferences on Artificial Intelligence

Layer-wise magnitude-based pruning (LMP) is a very popular method for deep neural network (DNN) compression. However, tuning the layer-specific thresholds is a difficult task, since the space of threshold candidates is exponentially large and the evaluation is very expensive. Previous methods are mainly by hand and require expertise. In this paper, we propose an automatic tuning approach based on optimization, named OLMP. The idea is to transform the threshold tuning problem into a constrained optimization problem (i.e., minimizing the size of the pruned model subject to a constraint on the accuracy loss), and then use powerful derivative-free optimization algorithms to solve it. To compress a trained DNN, OLMP is conducted within a new iterative pruning and adjusting pipeline. Empirical results show that OLMP can achieve the best pruning ratio on LeNet-style models (i.e., 114 times for LeNet-300-100 and 298 times for LeNet-5) compared with some state-ofthe-art DNN pruning methods, and can reduce the size of an AlexNet-style network up to 82 times without accuracy loss.
© 2018 International Joint Conferences on Artificial Intelligence. All right reserved.

其他

EI

[2017YFB1003102] ; [ZDSYS201703031748284] ; National Natural Science Foundation of China[61672478] ; National Natural Science Foundation of China[61603367] ; [2016QNRC001]

20184406016535

Artificial intelligence ; Arts computing ; Constrained optimization ; Iterative methods

Data Processing and Image Processing:723.2 ; Artificial Intelligence:723.4 ; Numerical Methods:921.6 ; Systems Science:961

EV Compendex

1.Anhui Province Key Lab of Big Data Analysis and Application, University of Science and Technology of China, Hefei; 230027, China
2.CERCIA, School of Computer Science, University of Birmingham, Birmingham; B15 2TT, United Kingdom
3.Shenzhen Key Lab of Computational Intelligence, Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen; 518055, China

Li, Guiying,Qian, Chao,Jiang, Chunhui,et al. Optimization based layer-wise magnitude-based pruning for DNN compression[C]:International Joint Conferences on Artificial Intelligence,2018:2383-2389.