Proximal policy optimization with model-based methods

Li, Shuailong1,2,3; Zhang, Wei1,2; Zhang, Huiwen4; Zhang, Xin1,2; Leng, Yuquan5,6

2022

10.3233/JIFS-211935

JOURNAL OF INTELLIGENT & FUZZY SYSTEMS   影响因子和分区

1064-1246

1875-8967

Model-free reinforcement learning methods have successfully been applied to practical applications such as decision-making problems in Atari games. However, these methods have inherent shortcomings, such as a high variance and low sample efficiency. To improve the policy performance and sample efficiency of model-free reinforcement learning, we propose proximal policy optimization with model-based methods (PPOMM), a fusion method of both model-based and model-free reinforcement learning. PPOMM not only considers the information of past experience but also the prediction information of the future state. PPOMM adds the information of the next state to the objective function of the proximal policy optimization (PPO) algorithm through a model-based method. This method uses two components to optimize the policy: the error of PPO and the error of model-based reinforcement learning. We use the latter to optimize a latent transition model and predict the information of the next state. For most games, this method outperforms the state-of-the-art PPO algorithm when we evaluate across 49 Atari games in the Arcade Learning Environment (ALE). The experimental results show that PPOMM performs better or the same as the original algorithm in 33 games.

Model-based reinforcement learning model-free reinforcement learning policy optimization method

SCI ; EI

英语

通讯

National Natural Science Foundation of China[52175272] ; State Key Laboratory of Robotics, China[2020-KF-22-03] ; StateKey Laboratory of Robotics Foundation[Y91Z0303] ; China Postdoctoral Science Foundation[2020M670814] ; Liaoning Provincial Natural Science Foundation[2020-MS-033]

Computer Science

Computer Science, Artificial Intelligence

WOS:000790690300042

IOS PRESS

20222012111244

Computer aided instruction ; Decision making ; Reinforcement learning

Artificial Intelligence:723.4 ; Computer Applications:723.5 ; Education:901.2 ; Management:912.2 ; Production Engineering:913.1

COMPUTER SCIENCE

Web of Science

1.Chinese Acad Sci, Shenyang Inst Automat, State Key Lab Robot, Shenyang, Peoples R China
2.Chinese Acad Sci, Inst Robot & Intelligent Mfg, Shenyang, Peoples R China
3.Univ Chinese Acad Sci, Beijing, Peoples R China
4.CVTE Res, Guangzhou, Peoples R China
5.Southern Univ Sci & Technol, Dept Mech & Energy Engn, Shenzhen Key Lab Biomimet Robot & Intelligent Sys, Shenzhen, Peoples R China
6.Southern Univ Sci & Technol, Guangdong Prov Key Lab Human Augmentat & Rehabil, Shenzhen, Peoples R China

Li, Shuailong,Zhang, Wei,Zhang, Huiwen,et al. Proximal policy optimization with model-based methods[J]. JOURNAL OF INTELLIGENT & FUZZY SYSTEMS,2022,42(6):5399-5410.

Li, Shuailong,Zhang, Wei,Zhang, Huiwen,Zhang, Xin,&Leng, Yuquan.(2022).Proximal policy optimization with model-based methods.JOURNAL OF INTELLIGENT & FUZZY SYSTEMS,42(6),5399-5410.

Li, Shuailong,et al."Proximal policy optimization with model-based methods".JOURNAL OF INTELLIGENT & FUZZY SYSTEMS 42.6(2022):5399-5410.