Log-normality and skewness of estimated state/action values in reinforcement learning

Zhang，Liangpeng1,2; Tang，Ke1,3; Yao，Xin2,3

2017

1049-5258

Advances in Neural Information Processing Systems

2017-December

1805-1815

Under/overestimation of state/action values are harmful for reinforcement learning agents. In this paper, we show that a state/action value estimated using the Bellman equation can be decomposed to a weighted sum of path-wise values that follow log-normal distributions. Since log-normal distributions are skewed, the distribution of estimated state/action values can also be skewed, leading to an imbalanced likelihood of under/overestimation. The degree of such imbalance can vary greatly among actions and policies within a single problem instance, making the agent prone to select actions/policies that have inferior expected return and higher likelihood of overestimation. We present a comprehensive analysis to such skewness, examine its factors and impacts through both theoretical and empirical results, and discuss the possible ways to reduce its undesirable effects.

其他

英语

2-s2.0-85047021495

Scopus

1.School of Computer Science and Technology,University of Science and Technology of China,China
2.University of Birmingham,United Kingdom
3.Shenzhen Key Lab of Computational Intelligence,Department of Computer Science and Engineering,Southern University of Science and Technology,China

Zhang，Liangpeng,Tang，Ke,Yao，Xin. Log-normality and skewness of estimated state/action values in reinforcement learning[C],2017:1805-1815.