Policy gradient adaptive dynamic programming for nonlinear discrete-time zero-sum games with unknown dynamics

Lin, Mingduo1; Zhao, Bo1,2; Liu, Derong3,4

2023

10.1007/s00500-023-07817-6

SOFT COMPUTING   影响因子和分区

1432-7643

1433-7479

A novel policy gradient (PG) adaptive dynamic programming method is developed to deal with nonlinear discrete-time zero-sum games with unknown dynamics. To facilitate the implementation, a policy iteration algorithm is established to approximate the iterative Q-function, as well as the control and disturbance policies via three neural network (NN) approximators, respectively. Then, the iterative Q-function is exploited to update the control and disturbance policies via PG method. To stabilize the training process and improve the data usage efficiency, the experience replay technique is applied to train the weight vectors of the three NNs by using mini-batch empirical data from replay memory. Furthermore, the convergence in terms of the iterative Q-function is proved. Simulation results of two numerical examples are provided to show the effectiveness of the proposed method.

Adaptive dynamic programming Reinforcement learning Neural networks Zero-sum games Policy gradient

SCI ; EI

英语

其他

Beijing Natural Science Foundation[4212038] ; National Natural Science Foundation of China["61973330","62073085"] ; Open Research Project of the State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences[20210108] ; Open Research Project of the Key Laboratory of Industrial Internet of Things & Networked Control, Ministry of Education[2021FF10]

Computer Science

Computer Science, Artificial Intelligence ; Computer Science, Interdisciplinary Applications

WOS:000915638100001

SPRINGER

20230313393107

Dynamic programming ; Game theory ; Iterative methods ; Neural networks ; Numerical methods

Artificial Intelligence:723.4 ; Optimization Techniques:921.5 ; Numerical Methods:921.6 ; Probability Theory:922.1

COMPUTER SCIENCE

Web of Science

1.Beijing Normal Univ, Sch Syst Sci, Beijing 100875, Peoples R China
2.Chongqing Univ Posts & Telecommun, Key Lab Ind Internet Things & Networked Control, Minist Educ, Chongqing 400065, Peoples R China
3.Southern Univ Sci & Technol, Dept Mech & Energy Engn, Shenzhen 518055, Peoples R China
4.Univ Illinois, Dept Elect & Comp Engn, Chicago, IL 60607 USA

Lin, Mingduo,Zhao, Bo,Liu, Derong. Policy gradient adaptive dynamic programming for nonlinear discrete-time zero-sum games with unknown dynamics[J]. SOFT COMPUTING,2023,27(9):5781-5795.

Lin, Mingduo,Zhao, Bo,&Liu, Derong.(2023).Policy gradient adaptive dynamic programming for nonlinear discrete-time zero-sum games with unknown dynamics.SOFT COMPUTING,27(9),5781-5795.

Lin, Mingduo,et al."Policy gradient adaptive dynamic programming for nonlinear discrete-time zero-sum games with unknown dynamics".SOFT COMPUTING 27.9(2023):5781-5795.