A Deep Reinforcement Learning-based Decentralized Hierarchical Motion Control Strategy for Multiple Amphibious Spherical Robot Systems with Tilting Thrusters

The variable operating conditions and hostile environments faced by underwater robots remains a challenge for motion control in unknown environments. In order to improve the capability of the amphibious spherical robot (ASR) in the unknown environments, a decentralized hierarchical deep reinforcement learning (DRL) motion control method based on deep deterministic policy gradient (DDPG) for multiple amphibious spherical robots system is proposed. In the low-level, a DDPG-based motion controller is trained under a compound rewarding to learn to set the configurations of the tilting angle and rotational speed of each thruster at a proper timescale. In the high-level, a planning controller consisting of different action networks is designed to generate a reasonable thrust target to guide the movement of the robot. Specifically, inspired by the artificial potential field method, the complex underwater motion can be decomposed into several simple virtual forces. Each action network is trained to learn to generate a virtual thrust target component for a specific action. By combining the outputs of several action networks, the distributed cooperative motion control for multi-robot systems can then be easily achieved. Finally, the motion control strategy is applied to the physical multi-ASR system, and the experiment results have shown satisfactory performance.