基于强化学习的无人机基站轨迹优化研究

TRAJECTORY OPTIMIZATION OF UAV BASE STATIONS BASED ON REINFORECEMENT LEARNING

刘真榕

11849160

硕士

信息与通信工程

贡毅

2020-05-29

2020-07-24

哈尔滨工业大学

深圳

无人机（Unmanned Aerial Vehicle, UAV）是由使用者远距离操控或嵌入式系统操纵驾驶的飞机。无人机因为其体积小，灵活性高，部署方便等特点被广泛的应用。无人机在无线通信领域的应用主要被分为两类：一方面，无人机被认为是接入蜂窝网络进行通信的新型空中用户。另一方面，无人机被用作新的空中通信平台，如基站和中继等，通过提供数据访问来协助地面无线通信。随着无人机制造成本的不断降低和通信设备的小型化，在无人机上安装小型基站或中继设备使飞行的空中平台能够辅助地面无线通信变得可行。这些基站适合安装在具有中等有效载荷的无人机上。与传统的地面通信基站相比，无人机作为基站提供通信服务有很多优势，比如更好的信道条件，更灵活的部署方式等等。无人机基站的轨迹优化是无人机通信系统中的一个重要问题，因为无人机的位置直接影响信道环境以及与地面用户的相对距离。在实际情况中，无人机在部署时很难精确地知道移动用户的确切位置。此外，由于无人机所能携带的能量有限，无人机通信系统的表现通常大打折扣。因此，本文在没有地面用户准确位置信息的情况下对无人机基站的轨迹优化问题进行研究分析，并在此基础上提出并解决基于能量效率最大化的无人机基站轨迹优化问题。本文首先研究了无人机通信系统并对无人机基站的轨迹优化问题进行建模。在研究了强化学习相关算法后，本文基于栅格法和Q-learning算法解决了在没有用户准确位置信息的情况下的无人机轨迹优化问题。在此基础上，我们提出了基于能量效率最大化的无人机基站轨迹优化问题。由于Q-learning算法的局限性，本文使用深度强化学习算法对该问题进行求解。仿真结果表明本文所提出的无人机轨迹优化算法能提高机载能源能量的利用效率。

The unmanned aerial vehicle (UAV) is a kind of aircraft which is controlled by users or embedded system. UAV is widely used because of its small size, high flexibility and convenient deployment. The application of UAV in wireless communication is divided into two categories: on the one hand, UAV is considered as a new type of air user accessing cellular network for communication. On the other hand, UAV is used as an air communication platform, such as base station and relay, to assist ground users by providing data access. With the continuous reduction of UAV manufacturing cost and the miniaturization of communication equipment, it is now feasible to install small base station or relay equipment on UAV so that the UAV can assist the wireless communication system. Compared with traditional terrestrial base station, UAV as a base station has many advantages, such as providing a better channel condition, a more flexible deployment and so on.The trajectory optimization of UAV is an important issue in the design of UAV communication system. In practice, it is difficult to know exact location of mobile users when UAV is deployed. In addition, due to the limited onboard energy, the performance of UAV communication system is usually greatly reduced. Therefore, this paper studied and analyzed the trajectory optimization of UAV base station without accurate user location and proposed and solved the trajectory optimization problem of UAV base station that maximize the energy efficiency. In this paper, the communication system of UAV is studied firstly, and the trajectory optimization of UAV base station is modeled, then the reinforcement learning algorithm is studied. Finally, the trajectory optimization problem of UAV without accurate user position is solved based on grid method and Q-learning algorithm. On this basis, we proposed the trajectory optimization of UAV base station that maximize the energy efficiency. Due to the limitations of Q-learning algorithm, this paper used deep reinforcement learning algorithm to solve the problem. The simulation results show that the proposed algorithm can improve the energy efficiency of UAV base station.

无人机基站 轨迹优化 能量效率 强化学习

unmanned aerial vehicle base station trajectory optimization energy efficiency reinforcement learning

中文

联合培养

南方科技大学

刘真榕. 基于强化学习的无人机基站轨迹优化研究[D]. 深圳. 哈尔滨工业大学,2020.