基于深度迁移学习的网络异常检测技术研究

RESEARCH ON NETWORK ANOMALY DETECTION BASED ON DEEP TRANSFER LEARNING

吕麒

11849323

硕士

计算机技术领域工程

汪漪

2020-05-29

2020-07-20

哈尔滨工业大学

深圳

网络攻击是当今信息化社会的一个严重问题，入侵检测系统是网络安全的重要保障。最近的研究将深度学习应用到网络入侵检测中，与基于规则签名和基于特征工程的传统机器学习算法相比有了很大的改进。但是这些深度学习方法很少考虑到异常样本的不平衡性，因此很难保证在异常样本很少的训练集上训练一个有效模型。同时由于网络流量在不同场景下的特征变化，在一个网络流数据集上训练得到的模型，很难直接用到新的场景中。生成式对抗网络（GAN）作为一种无监督深度学习方法，在许多领域取得很好的效果。为了解决以上问题，本文提出一种基于GAN的深度迁移学习网络异常检测框架，并在不同的公开数据集上进行测试，验证了算法的有效性。本文的主要研究工作和成果如下：针对网络流量数据中正常和异常样本的不平衡问题，首先利用卷积神经网络（CNN）和门控循环单元（GRU）分别捕获网络流的空间特征和时序特征，然后利用一种改进的GAN生成模拟异常样本，当训练集中异常样本不足时，使用生成的模拟异常样本进行补齐，从而达到样本占比的自平衡。在多个数据集上的实验表明，本文提出的方法在异常样本低于10%的数据集上的平均异常检测率和检测准确率为92.65%和90.11%，同时本文也对比分析了数据集中异常样本占比对深度模型的性能影响。为了解决网络环境下数据特征的变化和网络攻击新颖性带来的模式迁移问题，本文利用对抗式域适应的方法进行迁移学习，通过训练一个目标域特征提取器完成源域和目标域的特征对齐，使得在源域样本上训练好的模型能够方便的迁移到无标签，少数据的目标域中。在多个数据集上做的迁移学习实验结果表明，本文提出的方法在跨域的网络异常检测上相比不使用迁移学习的检测率和准确率分别提升了7.37%和8.43%。考虑到网络流数据不能直接输入神经网络中，为了避免复杂的特征工程，充分发挥神经网络强大的表征学习能力。本文实现一个基于字节解析的通用网络流处理工具，可以将PCAP形式的网络流数据快速处理成能够直接被输入到神经网络的数据形式。这为在实际场景中能够支持直接针对网络流量数据进行实时的异常检测提供一种有效的方法。

Network attack is a serious problem in information society nowdays.Intrusion detection system is an important guarantee for network security. Recent studies have applied deep learning to network intrusion detection, which has great improvement compared with traditional machine learning algorithms based on rule signatures and feature engineering. However, these methods take the imbalance of anomaly samples into account rarely. Therefore, it is difficult to ensure the effectiveness of the algorithm on the training set without enough anomaly samples. At the same time, due to the feature shift in the network traffic among different scenarios, it is difficult to apply the models trained on a network flow data set in a new scenario directly. Generative adversarial network (GAN) as an unsupervised deep learning method has achieved good results in many fields. In order to solve the above problems, this paper proposes a GAN-based deep transfer learning network anomaly detection framework. Evaluated on different public data sets, the algorithm achieves satisfied performance. The main research work and contributions of this paper are listed as follows.Aiming at solving the imbalance problem on network flow data, an improved GAN is used to generate simulated anomaly samples. Besides, the convolutional neural network (CNN) and Gate Recurrent Unit (GRU) are used to capture the spatio-temporal feature. When the anomaly samples in the training set are used up, the generated simulated anomaly samples will be used to fill in.The input of model can achieve the self-balance of the sample ratio by this way. Experiments on multiple data sets show that the method proposed in this paper can achieve an average anomaly detection rate of 92.65% and detection accuracy of 90.11% on data sets with anomaly sample of less than 10%. The effect of anomaly sample ratio in training data set for the performance of algorithm is also analyzed in this paper.In order to solve the problem of mode shift caused by the dynamic changes of data characteristics in the network environment and the novelty of network attacks, this paper proposes the adversarial domain adaptation. The feature between source domain and target domain is aligned by training a target domain feature extractor.The model trained on the source domain can be easily transferred to the unlabeled small volume target domain in this method. Tests on multiple data sets show that the average detection rate and accuracy can be improved by 7.37% and 8.43% respectively compared with model without transfer learning. The deep neural network can learn the feature from the raw data without complex feature project. However, the network flow data cannot be directly input into the neural network. This paper implements a universal network flow processing tool based on byte parsing. It can quickly process network flow data in the form of PCAP into a data format that can be input to the neural network directly. This provides an effective basic method for real-time anomaly detection for network flow data in actual scenarios.

网络异常检测 生成对抗网络 深度学习 迁移学习

network anomaly detection generative adversarial learning deep learning transfer learning

中文

联合培养

南方科技大学

吕麒. 基于深度迁移学习的网络异常检测技术研究[D]. 深圳. 哈尔滨工业大学,2020.