BiES: Adaptive Policy Optimization for Model-Based Offline Reinforcement Learning

Yang，Yijun1,2; Jiang，Jing1; Wang，Zhuowei1; Duan，Qiqi2; Shi，Yuhui2

10.1007/978-3-030-97546-3_46

2022

34th Australasian Joint Conference on Artificial Intelligence (AI)

0302-9743

1611-3349

978-3-030-97545-6

LECT NOTES ARTIF INT

13151 LNAI

570-581

FEB 02-04, 2022

Univ Technol Sydney,null,ELECTR NETWORK

GEWERBESTRASSE 11, CHAM, CH-6330, SWITZERLAND

SPRINGER INTERNATIONAL PUBLISHING AG

Offline reinforcement learning (RL) aims to train an agent solely using a dataset of historical interactions with the environments without any further costly or dangerous active exploration. Model-based RL (MbRL) usually achieves promising performance in offline RL due to its high sample-efficiency and compact modeling of a dynamic environment. However, it may suffer from the bias and error accumulation of the model predictions. Existing methods address this problem by adding a penalty term to the model reward but require careful hand-tuning of the penalty and its weight. Instead in this paper, we formulate the model-based offline RL as a bi-objective optimization where the first objective aims to maximize the model return and the second objective is adaptive to the learning dynamics of the RL policy. Thereby, we do not need to tune the penalty and its weight but can achieve a more advantageous trade-off between the final model return and model’s uncertainty. We develop an efficient and adaptive policy optimization algorithm equipped with evolution strategy to solve the bi-objective optimization, named as BiES. The experimental results on a D4RL benchmark show that our approach sets the new state of the art and significantly outperforms existing offline RL methods on long-horizon tasks.

Evolution strategy Multi-objective optimization Offline reinforcement learning

通讯

英语

CPCI-S ; EI

Computer Science

Computer Science, Artificial Intelligence

WOS:000787242700046

20221311865996

Economic and social effects ; Evolutionary algorithms ; Reinforcement learning

Artificial Intelligence:723.4 ; Optimization Techniques:921.5 ; Social Sciences:971

2-s2.0-85127216343

Scopus

1.AAII,University of Technology Sydney,Ultimo,2007,Australia
2.Department of Computer Science and Engineering,Southern University of Science and Technology,Shenzhen,518055,China

Yang，Yijun,Jiang，Jing,Wang，Zhuowei,et al. BiES: Adaptive Policy Optimization for Model-Based Offline Reinforcement Learning[C]. GEWERBESTRASSE 11, CHAM, CH-6330, SWITZERLAND:SPRINGER INTERNATIONAL PUBLISHING AG,2022:570-581.