马尔可夫决策过程在无线缓存网络中的应用

APPLICATION OF MARKOV DECISION PROCESS IN WIRELESS CACHING NETWORKS

吕博杰

11849150

硕士

信息与通信工程

王锐

2020-05-29

2020-07-23

哈尔滨工业大学

深圳

随着无线通信技术的发展，无线传输速率越来越快。人们对无线数据传输的需求也日益增长。与此同时，以内容为中心的数据（视频、音频等）逐渐成为无线数据传输的主流。无线缓存技术就是把这些以内容为中心的数据储存在网络边缘的缓存节点中，从而提高网络的整体性能。本文研究了在缓存节点（存储空间受限）的帮助下，单个蜂窝网络中下行文件传输的调度。具体来说，用户随机地向基站提出文件访问的请求。当文件访问请求到达的时候，基站将该文件多播到发出请求的用户和一些选定的缓存节点。当下一次访问相同文件的用户落在这些缓存节点的覆盖范围内的时候，缓存节点可以代替基站承接文件传输任务。在上述文件有限的生命周期内，由于缓存节点的储存空间受限，文件的放置（到缓存节点）和文件的传输（到用户）需要联合优化。在文件的生命周期中，基站对每次多播传输资源的分配（多播功率和传输符号数量）被建模成一个具有随机阶段数量的动态规划（Dynamic Programming）问题。此类问题无法用现有方法解决，因此，通过将原始问题转换为具有固定阶段数的等效马尔可夫决策过程（Markov Decision Process），本文开发了一种渐近最优的框架来解决该问题。由于动态规划问题常常遇到“维度爆炸”（Curse of Dimensionality）问题，本文提出了一种新颖的近似方法来解决“维度爆炸”问题。具体来说，本文首先设计了一个灵活的框架来降低计算值函数所需要的阶段数，然后用线性近似的方法来近似值函数。其中，本文推导了近似值函数（Approximate Value Function）的解析表达式，而且还推导出准确值函数和近似值函数之间误差的解析上界。基于近似值函数的表达式，本文给出了一种低复杂度的在线资源分配算法。最后，由于近似值函数取决于一些系统参数的统计信息（用户的地理分布、文件的受欢迎程度等）。当这些统计数据未知的时候，本文提出了一种强化学习算法来学习这些参数，从而帮助基站更好地分配传输资源。通过数值仿真表明，与一些基准方案相比，本文所提出的基于近似值函数的低复杂度算法可以显著降低基站的平均传输成本。

With the development of wireless communication technology, wireless transmission rate is getting faster. People's demand for wireless data transmission is also increasing. At the same time, content-centric data (video, audio, etc.) has gradually become the mainstream of wireless data transmission. Wireless cache technology is to store these content-centric data in cache nodes at the edge of the network, thereby improving the overall performance of the network.In this paper, the scheduling of downlink file transmission in one cell with the assistance of cache nodes with finite cache space is studied. Specifically, requesting users arrive randomly and the base station (BS) reactively multicasts files to the requesting users and selected cache nodes. The latter can offload the traffic in their coverage areas from the BS. We consider the joint optimization of the abovementioned file placement and delivery within a finite lifetime subject to the cache space constraint. Within the lifetime, the allocation of multicast power and symbol number for each file transmission at the BS is formulated as a dynamic programming problem with a random stage number. Note that there are no existing solutions to this problem. We develop an asymptotically optimal solution frameworkby transforming the original problem to an equivalent finite-horizon Markov decision process (MDP) with a fixed stage number. A novel approximation approach is then proposed to address the curse of dimensionality, where the analytical expressions of approximate value functions are provided. We also derive analytical bounds on the exact value function and approximation error. Based on the expression of approximate value function, this paper presents a low complexity online resource allocation algorithm.The approximate value functions depend on some system statistics, e.g., requesting users’ distribution. One reinforcement learning algorithm is proposed for the scenario where these statistics are unknown.Numerical simulations show that the low-complexity algorithm based on the approximation function proposed in this paper can significantly reduce the average transmission cost of the base station compared with some benchmark schemes.

无线缓存网络 马尔可夫决策过程 强化学习 近似算法

wireless caching networks markov decision process reinforcement learning approximate algorithm

中文

联合培养

南方科技大学

吕博杰. 马尔可夫决策过程在无线缓存网络中的应用[D]. 深圳. 哈尔滨工业大学,2020.