Adaptive Policy Learning for Offline-to-Online Reinforcement Learning

Zheng，Han1; Luo，Xufang2; Wei，Pengfei3; Song，Xuan4; Li，Dongsheng2; Jiang，Jing1

2023-06-27

37th AAAI Conference on Artificial Intelligence (AAAI) / 35th Conference on Innovative Applications of Artificial Intelligence / 13th Symposium on Educational Advances in Artificial Intelligence

2159-5399

2374-3468

*****************

Proceedings of the 37th AAAI Conference on Artificial Intelligence, AAAI 2023

37

11372-11380

FEB 07-14, 2023

null,Washington,DC

2275 E BAYSHORE RD, STE 160, PALO ALTO, CA 94303 USA

ASSOC ADVANCEMENT ARTIFICIAL INTELLIGENCE

Conventional reinforcement learning (RL) needs an environment to collect fresh data, which is impractical when online interactions are costly. Offline RL provides an alternative solution by directly learning from the previously collected dataset. However, it will yield unsatisfactory performance if the quality of the offline datasets is poor. In this paper, we consider an offline-to-online setting where the agent is first learned from the offline dataset and then trained online, and propose a framework called Adaptive Policy Learning for effectively taking advantage of offline and online data. Specifically, we explicitly consider the difference between the online and offline data and apply an adaptive update scheme accordingly, that is, a pessimistic update strategy for the offline dataset and an optimistic/greedy update scheme for the online dataset. Such a simple and effective method provides a way to mix the offline and online RL and achieve the best of both worlds. We further provide two detailed algorithms for implementing the framework through embedding value or policy-based RL algorithms into it. Finally, we conduct extensive experiments on popular continuous control tasks, and results show that our algorithm can learn the expert policy with high sample efficiency even when the quality of offline dataset is poor, e.g., random dataset.

其他

英语

EI ; CPCI-S

Computer Science ; Mathematics

Computer Science, Artificial Intelligence ; Computer Science, Theory & Methods ; Mathematics, Applied

WOS:001243747800120

20233414600821

E-learning

Artificial Intelligence:723.4

2-s2.0-85162680903

Scopus

1.University of Technology Sydney,Australia
2.Microsoft Research Asia,China
3.National University of Singapore,Singapore
4.Southern University of Science and Technology,China

Zheng，Han,Luo，Xufang,Wei，Pengfei,et al. Adaptive Policy Learning for Offline-to-Online Reinforcement Learning[C]. 2275 E BAYSHORE RD, STE 160, PALO ALTO, CA 94303 USA:ASSOC ADVANCEMENT ARTIFICIAL INTELLIGENCE,2023:11372-11380.