Analysis of Q-learning with adaptation and momentum restart for gradient descent

Weng，Bowen1; Xiong，Huaqing1; Liang，Yingbin1; Zhang，Wei2

2020

1045-0823

IJCAI International Joint Conference on Artificial Intelligence

2021-January

3051-3057

Existing convergence analyses of Q-learning mostly focus on the vanilla stochastic gradient descent (SGD) type of updates. Despite the Adaptive Moment Estimation (Adam) has been commonly used for practical Q-learning algorithms, there has not been any convergence guarantee provided for Q-learning with such type of updates. In this paper, we first characterize the convergence rate for Q-AMSGrad, which is the Q-learning algorithm with AMSGrad update (a commonly adopted alternative of Adam for theoretical analysis). To further improve the performance, we propose to incorporate the momentum restart scheme to Q-AMSGrad, resulting in the so-called Q-AMSGradR algorithm. The convergence rate of Q-AMSGradR is also established. Our experiments on a linear quadratic regulator problem show that the two proposed Q-learning algorithms outperform the vanilla Q-learning with SGD updates. The two algorithms also exhibit significantly better performance than the DQN learning method over a batch of Atari 2600 games.

通讯

英语

EI

20205009609613

Learning systems ; Gradient methods ; Learning algorithms

Machine Learning:723.4.2 ; Control Systems:731.1 ; Numerical Methods:921.6 ; Systems Science:961

2-s2.0-85095333017

Scopus

1.Electrical and Computer Engineering,Ohio State University,Columbus,United States
2.Mechanical and Energy Engineering,Southern University of Science and Technology,China

Weng，Bowen,Xiong，Huaqing,Liang，Yingbin,et al. Analysis of Q-learning with adaptation and momentum restart for gradient descent[C],2020:3051-3057.