BSO-ES: A Hybrid Direct Policy Search Algorithm for Reinforcement Learning

Zhang，Youkui; Zhang，Liang; Duan，Qiqi; Shi，Yuhui

10.1109/SSCI50451.2021.9660122

2021

IEEE Symposium Series on Computational Intelligence (IEEE SSCI)

978-1-7281-9049-5

2021 IEEE Symposium Series on Computational Intelligence, SSCI 2021 - Proceedings

01-06

5-7 Dec. 2021

Orlando, FL, USA

345 E 47TH ST, NEW YORK, NY 10017 USA

IEEE

Recently, it has been demonstrated that evolution strategies (ES), a class of blackbox optimization algorithms, can achieve competitive performance on many reinformance learning (RL) tasks, compared to deep RL algorithms. However, ES can be seen as a randomized local search algorithm based on stochastic finite difference and it is often hard to escape from deep or deceptive local optimum, which means that its exploration ability can be further improved. Brain storm optimization (BSO) is a swarm intelligence algorithm with tradeoff between exploration and exploitation via clustering. To enjoy best of both worlds, this paper proposes a hybrid direct policy search algorithm (BSO-ES) for RL tasks within a distributed computing framework. Specifically, we use BSO as a global searcher to explore the parameter space and ES as a local searcher to further fine-tune policies. We maintain multiple parallel policies at the same time and exchange preferable policies periodically. Experimental results on six challenging continuous control tasks from MuJoCo show that our proposed algorithm is superior to or competitive with OpenAI-ES. In particular, our algorithm has less failures to reach a fixed reward threshold.

Brain storm optimization Direct policy search Evolution strategy Reinforcement learning

第一

英语

CPCI-S ; EI

Shenzhen Fundamental Research Program[JCYJ20200109141235597]

Computer Science ; Engineering ; Operations Research & Management Science ; Mathematics

Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic ; Operations Research & Management Science ; Mathematics, Applied

WOS:000824464300299

20221011761149

Clustering algorithms ; Evolutionary algorithms ; Learning algorithms ; Local search (optimization) ; Stochastic systems ; Storms

Precipitation:443.3 ; Artificial Intelligence:723.4 ; Machine Learning:723.4.2 ; Control Systems:731.1 ; Information Sources and Analysis:903.1 ; Optimization Techniques:921.5 ; Systems Science:961

2-s2.0-85125765568

Scopus

Southern University of Science and Technology,Department of Computer Science and Engineering,Shenzhen,China

Zhang，Youkui,Zhang，Liang,Duan，Qiqi,et al. BSO-ES: A Hybrid Direct Policy Search Algorithm for Reinforcement Learning[C]. 345 E 47TH ST, NEW YORK, NY 10017 USA:IEEE,2021:01-06.