Accelerating multi-objective neural architecture search by random-weight evaluation

Hu, Shengran1; Cheng, Ran1; He, Cheng1; Lu, Zhichao1; Wang, Jing2; Zhang, Miao2

2021-12-01

10.1007/s40747-021-00594-5

Complex & Intelligent Systems   影响因子和分区

2199-4536

2198-6053

For the goal of automated design of high-performance deep convolutional neural networks (CNNs), neural architecture search (NAS) methodology is becoming increasingly important for both academia and industries. Due to the costly stochastic gradient descent training of CNNs for performance evaluation, most existing NAS methods are computationally expensive for real-world deployments. To address this issue, we first introduce a new performance estimation metric, named random-weight evaluation (RWE) to quantify the quality of CNNs in a cost-efficient manner. Instead of fully training the entire CNN, the RWE only trains its last layer and leaves the remainders with randomly initialized weights, which results in a single network evaluation in seconds. Second, a complexity metric is adopted for multi-objective NAS to balance the model size and performance. Overall, our proposed method obtains a set of efficient models with state-of-the-art performance in two real-world search spaces. Then the results obtained on the CIFAR-10 dataset are transferred to the ImageNet dataset to validate the practicality of the proposed algorithm. Moreover, ablation studies on NAS-Bench-301 datasets reveal the effectiveness of the proposed RWE in estimating the performance compared to existing methods.

Neural architecture search Efficient performance estimation Multi-objective optimization Evolutionary algorithm

SCI

英语

第一 ; 通讯

National Natural Science Foundation of China[61903178,61906081,"U20A20306"] ; Shenzhen Science and Technology Program[RCBS20200714114817264] ; Guangdong Provincial Key Laboratory[2020B121201001]

Computer Science

Computer Science, Artificial Intelligence

WOS:000726263100001

SPRINGER HEIDELBERG

Web of Science

1.Southern Univ Sci & Technol, Dept Comp Sci & Engn, Guangdong Key Lab Brain Inspired Intelligent Comp, Shenzhen 518055, Peoples R China
2.Shanghai Aircraft Design & Res Inst, Shanghai 200135, Peoples R China

Hu, Shengran,Cheng, Ran,He, Cheng,et al. Accelerating multi-objective neural architecture search by random-weight evaluation[J]. Complex & Intelligent Systems,2021,9(2).

Hu, Shengran,Cheng, Ran,He, Cheng,Lu, Zhichao,Wang, Jing,&Zhang, Miao.(2021).Accelerating multi-objective neural architecture search by random-weight evaluation.Complex & Intelligent Systems,9(2).

Hu, Shengran,et al."Accelerating multi-objective neural architecture search by random-weight evaluation".Complex & Intelligent Systems 9.2(2021).