Improving Zero-Shot Coordination with Diversely Rewarded Partner Agents

Peilin Wu1; Zhenhua Yang1; Peng Yang1,2

10.1109/IJCNN60899.2024.10649945

2024-07-05

2161-4393

979-8-3503-5932-9

2024 International Joint Conference on Neural Networks (IJCNN)

30 June-5 July 2024

Yokohama, Japan

Zero-shot coordination studies the training of well-generalizing human-AI coordination agents in the scenario where human data is unavailable. To obtain a coordination agent generalize to unseen humans, prevailing methods generate a population of partner agents as proxy models of human partners and then train a coordination agent with these partner agents. Constructed partner agents are expected to be as diverse as possible to cover a wide range of human behaviors, preventing a distribution shift between training and testing stages. Recent works concentrate on studying effective methods of creating a group of high-reward while diverse partner agents to model unseen human partners. However, the resulting high-reward partner agents do not accurately reflect real-world situations, considering that human decisions are not always optimal and may sometimes even hinder the progression of coordination. Therefore, these studies still struggle to capture the potential characteristics of human partners. In this work, reinforcement learning (RL) and supervised learning (SL) are integrated to train a reward-conditioned policy. By conditioned on different desired rewards, a reward-conditioned policy simulates both low-reward and high-reward partners. Additionally, a reward-bucketed replay buffer and curriculum learning are applied to enhance reward diversity and boost the training of coordination agents. Experiments demonstrate that the proposed reward-conditioned policy is capable of generating agents with different rewards. Moreover, the zero-shot coordination performance of agents trained with these partners surpasses previous methods in the majority of scenarios within the Overcooked human-AI coordination benchmark.

第一

1.Department of Computer Science and Engineering, Guangdong Provincial Key Laboratory of Brain-Inspired Intelligent Computation, Southern University of Science and Technology, Shenzhen, China
2.Department of Statistics and Data Science, Research Institute of Trustworthy Autonomous Systems, Southern University of Science and Technology, Shenzhen, China

Peilin Wu,Zhenhua Yang,Peng Yang. Improving Zero-Shot Coordination with Diversely Rewarded Partner Agents[C],2024.