An Offline-Transfer-Online Framework for Cloud-Edge Collaborative Distributed Reinforcement Learning

Zeng，Tianyu1; Zhang，Xiaoxi1; Duan，Jingpu2; Yu，Chao1; Wu，Chuan3; Chen，Xu1

2024

10.1109/TPDS.2024.3360438

IEEE Transactions on Parallel and Distributed Systems   影响因子和分区

1045-9219

1558-2183

Recent advances in deep reinforcement learning (DRL) have made it possible to train various powerful agents to perform complex tasks in real-time environments. With the next-generation communication technologies, making cloud-edge collaborative artificial intelligence service with evolved DRL agents can be a significant scenario. However, agents with different algorithms and architectures in the same DRL scenario may not be compatible, and training them is either time-consuming or resource-demanding. In this paper, we design a novel cloud-edge collaborative DRL training framework, named Offline-Transfer-Online, which is a new approach that can speed up the convergence of online DRL agents at the edge by interacting with offline agents in the cloud, with the minimum data interchanged and without relying on high-quality offline datasets. Therein, we propose a novel algorithm-independent knowledge distillation algorithm for online RL agents, by leveraging pre-trained models and the interface between agents and the environment to transfer distilled knowledge among multiple heterogeneous agents efficiently. Extensive experiments show that our algorithm can accelerate the convergence of various online agents in a double to decuple speed, with comparable reward achieved in different environments.

Cloud computing cloud-edge collaborative networks Collaboration deep reinforcement learning Distributed training Federated learning Offline-Transfer-Online Real-time systems Solid modeling Training Transfer learning

SCI ; EI

英语

其他

COMPUTER SCIENCE

2-s2.0-85184333067

Scopus

1.School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China
2.Institute of Future Networks, Southern University of Science and Technology, China
3.Department of Computer Science, University of Hong Kong, Hong Kong, China

Zeng，Tianyu,Zhang，Xiaoxi,Duan，Jingpu,et al. An Offline-Transfer-Online Framework for Cloud-Edge Collaborative Distributed Reinforcement Learning[J]. IEEE Transactions on Parallel and Distributed Systems,2024,35(5):720-731.

Zeng，Tianyu,Zhang，Xiaoxi,Duan，Jingpu,Yu，Chao,Wu，Chuan,&Chen，Xu.(2024).An Offline-Transfer-Online Framework for Cloud-Edge Collaborative Distributed Reinforcement Learning.IEEE Transactions on Parallel and Distributed Systems,35(5),720-731.

Zeng，Tianyu,et al."An Offline-Transfer-Online Framework for Cloud-Edge Collaborative Distributed Reinforcement Learning".IEEE Transactions on Parallel and Distributed Systems 35.5(2024):720-731.