Marten: A Built-in Security DRL-Based Congestion Control Framework by Polishing the Expert

Zhiyuan Pan1; Jianer Zhou1; Xinyi Qiu2; Weichao Li2; Heng Pan3; Wei Zhang3

10.1109/INFOCOM53939.2023.10228952

2023

0743-166X

979-8-3503-3415-9

IEEE INFOCOM 2023 - IEEE Conference on Computer Communications

2023-May

1-10

17-20 May 2023

New York City, NY, USA

Deep reinforcement learning (DRL) has been proved to be an effective method to improve the congestion control algorithms (CCAs). However, the lack of training data and training scale affect the effectiveness of DRL model. Combining rule-based CCAs (such as BBR) as a guide for DRL is an effective way to improve learning-based CCAs. By experiment measurement, we find that the rule-based CCAs limit the action exploration and even cause DRL’s excessive dependence to gain higher DRL’s reward gain. To overcome the constraints, we propose Marten, a framework which improves the effectiveness of rule-based CCAs for DRL. Marten uses entropy as the degree of exploration and uses it to expand the exploration of DRL. Furthermore, Marten introduces the shielding mechanism to avoid wrong DRL actions. We have implemented Marten in both simulation platform OpenAI Gym and deployment platform QUIC. The experimental results in production network demonstrate Marten can improve throughput by 0.36% and reduce latency by 14.89% on average compared with Eagle, and improve throughput by 2.79% and reduce latency by 11.73% on average compared with BBR.

Congestion Control Machine Learning Deep Reinforcement Learning QUIC

第一

EI

20233814771521

Deep learning ; Learning systems ; Simulation platform ; Throughput

Ergonomics and Human Factors Engineering:461.4 ; Artificial Intelligence:723.4 ; Computer Applications:723.5

IEEE

1.Southern University of Science and Technology, China
2.Peng Cheng Laboratory, China
3.ICT, CAS, China

Zhiyuan Pan,Jianer Zhou,Xinyi Qiu,et al. Marten: A Built-in Security DRL-Based Congestion Control Framework by Polishing the Expert[C],2023:1-10.