Theoretical Analysis of Value-Iteration-Based Q-Learning with Approximation Errors

Zhantao Liang1; Mingming Ha2; Derong Liu3

10.1109/ICIST55546.2022.9926794

2022

2164-4357

978-1-6654-9738-1

2022 12th International Conference on Information Science and Technology (ICIST)

120-125

14-16 Oct. 2022

Kaifeng, China

In this paper, the value-iteration-based Q-Iearning algorithm with approximation errors is analyzed theoretically. First, based on an upper bound of the approximation errors caused by the Q-function approximator, we get the lower and upper bound functions of the iterative Q-function, which proves that the limit of the approximate Q-function sequence is bounded. Then, we develop a stability condition for the termination of the iterative algorithm, for ensuring that the current control policy derived from the resulting approximate Q-function is stabilizing. Also, we establish an upper bound function of the approximation errors, which is caused by the policy function approximator, to guarantee that the approximate control policy is stabilizing. Finally, the numerical results verifies the theoretical results with a simulation example.

Adaptive dynamic programming Q-Iearning value iteration asymptotic stability

其他

IEEE

1.School of Automation, Guangdong University of Technology, Guangzhou, China
2.School of Automation and Electrical Engineering, University of Science and Technology Beijing, Beijing, China
3.Institute of Control Science and Technology, Southern University of Science and Technology, Shenzhen, China

Zhantao Liang,Mingming Ha,Derong Liu. Theoretical Analysis of Value-Iteration-Based Q-Learning with Approximation Errors[C],2022:120-125.