LSBO-NAS: Latent Space Bayesian Optimization for Neural Architecture Search

Xuan Rao1; Songyi Xiao1; Jiaxin Li1; Qiuye Wu1; Bo Zhao2; Derong Liu3

10.1109/ICCR55715.2022.10053904

2022

978-1-6654-8642-2

2022 4th International Conference on Control and Robotics (ICCR)

22-27

2-4 Dec. 2022

Guangzhou, China

From the perspective of data stream, neural architecture search (NAS) can be formulated as a graph optimization problem. However, many state-of-the-art black-box optimization algorithms, such as Bayesian optimization and simulated annealing, operate in continuous space primarily, which does not match the NAS optimization due to the discreteness of graph structures. To tackle this problem, the latent space Bayesian optimization NAS (LSBO-NAS) algorithm is developed in this paper. In LSBO-NAS, the neural architectures are represented as sequences, and a variational auto-encoder (VAE) is trained to convert the discrete search space of NAS into a continuous latent space by learning the continuous representation of neural architectures. Hereafter, a Bayesian optimization (BO) algorithm, i.e., the tree-structure parzen estimator (TPE) algorithm, is developed to obtain admirable neural architectures. The optimization loop of LSBO-NAS consists of two stages. In the first stage, the BO algorithm generates a preferable architecture representation according to its search strategy. In the second stage, the decoder of VAE decodes the representation into a discrete neural architecture, whose performance evaluation is regarded as the feedback signal for the BO algorithm. The effectiveness of the developed LSBO-NAS is demonstrated on the NAS-Bench-301 benchmark, where the LSBO-NAS achieves a better performance than several NAS baselines.

Neural architecture search deep generative model variational auto-encoder Bayesian optimization unsupervised representation learning

其他

IEEE

1.School of Automation, Guangdong University of Technology, Guangzhou, China
2.School of Systems Science, Beijing Normal University, Beijing, China
3.Institute of Control Science and Technology, Southern University of Science and Technology, Shenzhen, China

Xuan Rao,Songyi Xiao,Jiaxin Li,et al. LSBO-NAS: Latent Space Bayesian Optimization for Neural Architecture Search[C],2022:22-27.