基于时空特征分解的时空序列预测

伴随着人类社会智能化的进程，时空序列数据频繁的出现在人们的日常生活中。然而如何利用时空序列数据来对未来的趋势进行预测，从而辅助人们的决策，也就成为了当下的研究热点。时空序列预测在多个领域中都有广泛的应用，比如金融分析、智慧城市、气象监测等等。与此同时，时空类数据通常具有复杂性、随机性、空间相关性等特点，给模型的预测带来了不少困难。因此，如何进行精确的时空序列预测仍然是目前的一个难题。本文基于深度学习技术，分别根据交通流量数据和基站流量数据的特点，设计了两个模型能有效ᨀ取数据中的时空特征并给出准确的预测。本文的主要工作如下：（1）出了一种基于 Transformer 框架的交通流量预测模型。针对空间特征建模，我们结合图卷积网络和空间自注意力机制构建了一套融合静态空间结构和动态空间结构的空间特征建模单元。同时利用 Gumbel­ Softmax 函数构建了可求导的稀疏邻接矩阵来减轻以往工作中邻接矩阵过于平滑的问题。针对时序建模，我们利用时序自注意力机制和时序分解模块来提取数据中不同程度的时序特征，并结合时序融合模块构建出了时间维度的特征金字塔。最后，通过解码器模块来生成最终的预测。在四个数据集上进行了对比实验，结果表明我们的模型取得了较好的结果。(2)针对基站时空数据的特点，我们提出了一种基于图注意力网络的基站流量预测模型。我们利用门控循环网络对辅助特征进行融合，并结合注意力机制来抽取和预测目标有关的时序特征。鉴于基站数据地理强相关性的特点，我们将静态图结构和配置信息引入到模型中，利用SDNE网络将基站节点的距离矩阵转换为隐空间表示并嵌入到图注意力网络中，同时加入基站的配置信息。我们将图注意力网络和时序自注意力模块并行处理，用于同时提取时空特征。在运营商提供的数据集上进行了实验，结果显示我们的方法取得了较低的误差值，证实了模型的有效性。

With the process of intelligence in human society, spatiotemporal sequence data frequently appears in people's daily lives. However, how to use spatiotemporal sequence data to predict future trends and assist people's decision-making has become a current research hotspot. Spatiotemporal sequence prediction has wide applications in various fields, such as financial analysis, smart cities, and weather monitoring. At the same time, spatiotemporal data usually has characteristics such as complexity, randomness, and spatial correlation, which bring many difficulties to the model's prediction. Therefore, how to achieve accurate spatiotemporal sequence prediction is still a challenge. Based on deep learning technology, this paper designs two models according to the characteristics of traffic flow data and base station flow data, respectively, which can effectively extract the spatiotemporal features of the data and provide accurate predictions. The main work of this paper is as follows: (1) We propose a traffic flow prediction model based on the Transformer framework. For spatial feature modeling, we combine graph convolutional networks and spatial self-attention mechanisms to construct a spatial feature modeling unit that integrates static and dynamic spatial structures. Meanwhile, we use Gumbel-Softmax to construct a differentiable sparse adjacency matrix to alleviate the problem of overly smooth adjacency matrices in previous works. For temporal modeling, we use temporal self-attention mechanisms and temporal decomposition modules to extract temporal features of different degrees from the data, and combine them with a temporal fusion module to construct a feature pyramid in the time dimension. Finally, the decoder module is used to generate the final prediction. Comparative experiments on four datasets show that our model achieves good results. (2) To address the characteristics of spatiotemporal data in base stations, we propose a base station flow prediction model based on graph attention networks. We use gated recurrent networks to fuse auxiliary features and combine attention mechanisms to extract and predict target-related temporal features. Given the strong geographic correlation in base station data, we introduce static graph structures and configuration information into the model, use the SDNE network to convert the distance matrix of base station nodes into a latent space representation and embed it into the graph attention network, and include the configuration information of base stations. We parallelly process the graph attention network and temporal self-attention module to extract spatiotemporal features simultaneously. Experimental results on a dataset provided by an operator show that our method achieves lower error values, demonstrating the effectiveness of the model.

[1] SALINAS D, FLUNKERT V, GASTHAUS J, et al. DeepAR: Probabilistic forecasting with autoregressive recurrent networks[J]. International Journal of Forecasting, 2020, 36(3): 1181­ 1191.
[2] TRIEBE O, HEWAMALAGE H, PILYUGINA P, et al. NeuralProphet: Explainable Forecast­ ing at Scale[A]. 2021. arXiv: 2111.15397.
[3] LIM B, ARIK S Ö, LOEFF N, et al. Temporal fusion transformers for interpretable multi­ horizon time series forecasting[J]. International Journal of Forecasting, 2021, 37(4): 1748­ 1764.
[4] XIAO H, SUN H, RAN B, et al. Fuzzy­neural network traffic prediction framework with wavelet decomposition[J]. Transportation research record, 2003, 1836(1): 16­20.
[5] JEONG Y S, BYON Y J, CASTRO­NETO M M, et al. Supervised weighting­online learning algorithm for short­term traffic flow prediction[J]. IEEE Transactions on Intelligent Transporta­ tion Systems, 2013, 14(4): 1700­1707.
[6] SUN Y, LENG B, GUAN W. A novel wavelet­SVM short­time passenger flow prediction in Beijing subway system[J]. Neurocomputing, 2015, 166: 109­121.
[7] BILLAH B, KING M L, SNYDER R D, et al. Exponential smoothing model selection for forecasting[J]. International journal of forecasting, 2006, 22(2): 239­247.
[8] ZHANG G P. Time series forecasting using a hybrid ARIMA and neural network model[J]. Neurocomputing, 2003, 50: 159­175.
[9] ZIVOT E, WANG J. Vector autoregressive models for multivariate time series[J]. Modeling financial time series with S­PLUS®, 2006: 385­429.
[10] LECUN Y, BENGIO Y, HINTON G. Deep learning[J]. nature, 2015, 521(7553): 436­444.
[11] BAI S, KOLTER J Z, KOLTUN V. An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling[A]. 2018. arXiv: 1803.01271.
[12] HOCHREITER S, SCHMIDHUBER J. Long short­term memory[J]. Neural computation, 1997, 9(8): 1735­1780.
[13] LAI G, CHANG W C, YANG Y, et al. Modeling long­and short­term temporal patterns with deep neural networks[C]//The 41st international ACM SIGIR conference on research & devel­ opment in information retrieval. 2018: 95­104.
[14] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[J]. Advances in neural information processing systems, 2017, 30.
[15] LI S, JIN X, XUAN Y, et al. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting[J]. Advances in neural information processing systems, 2019, 32.
[16] ZHOU H, ZHANG S, PENG J, et al. Informer: Beyond efficient transformer for long sequence time­series forecasting[C]//Proceedings of the AAAI conference on artificial intelligence: vol­ ume 35. 2021: 11106­11115.
[17] SUTSKEVER I, VINYALS O, LE Q V. Sequence to sequence learning with neural networks [J]. Advances in neural information processing systems, 2014, 27.
[18] WU H, XU J, WANG J, et al. Autoformer: Decomposition transformers with auto­correlation for long­term series forecasting[J]. Advances in Neural Information Processing Systems, 2021, 34: 22419­22430.
[19] JIANG W, LUO J. Graph neural network for traffic forecasting: A survey[J]. Expert Systems with Applications, 2022: 117921.
[20] DIAO Z, WANG X, ZHANG D, et al. Dynamic spatial­temporal graph convolutional neural networks for traffic forecasting[C]//Proceedings of the AAAI conference on artificial intelli­ gence: volume 33. 2019: 890­897.
[21] ZHANG Q, CHANG J, MENG G, et al. Spatio­temporal graph structure learning for traf­ fic forecasting[C]//Proceedings of the AAAI conference on artificial intelligence: volume 34. 2020: 1177­1185.
[22] WU Z, PAN S, LONG G, et al. Connecting the dots: Multivariate time series forecasting with graph neural networks[C]//Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. 2020: 753­763.
[23] CHEN W, CHEN L, XIE Y, et al. Multi­range attentive bicomponent graph convolutional net­ work for traffic forecasting[C]//Proceedings of the AAAI conference on artificial intelligence: volume 34. 2020: 3529­3536.
[24] RAO X, WANG H, ZHANG L, et al. Fogs: First­order gradient supervision with learning­ based graph for traffic flow forecasting[C]//Proceedings of International Joint Conference on Artificial Intelligence, IJCAI. 2022.
[25] LAN S, MA Y, HUANG W, et al. Dstagnn: Dynamic spatial­temporal aware graph neural network for traffic flow forecasting[C]//International Conference on Machine Learning. PMLR, 2022: 11906­11917.
[26] YU B, YIN H, ZHU Z. Spatio­temporal graph convolutional networks: a deep learning frame­ work for traffic forecasting[C]//Proceedings of the 27th International Joint Conference on Ar­ tificial Intelligence. 2018: 3634­3640.
[27] LI Y, YU R, SHAHABI C, et al. Diffusion Convolutional Recurrent Neural Network: Data­ Driven Traffic Forecasting[C]//International Conference on Learning Representations. 2018.
[28] WU Z, PAN S, LONG G, et al. Graph wavenet for deep spatial­temporal graph modeling[C]// Proceedings of the 28th International Joint Conference on Artificial Intelligence. 2019: 1907­ 1913.
[29] BAI L, YAO L, LI C, et al. Adaptive graph convolutional recurrent network for traffic fore­ casting[J]. Advances in neural information processing systems, 2020, 33: 17804­17815.
[30] LIU S, YU H, LIAO C, et al. Pyraformer: Low­complexity pyramidal attention for long­rangetime series modeling and forecasting[C]//International conference on learning representations. 2021.
[31] ZHOU T, MA Z, WEN Q, et al. Fedformer: Frequency enhanced decomposed transformer for long­term series forecasting[C]//International Conference on Machine Learning. PMLR, 2022: 27268­27286.
[32] WEN Q, SUN L, SONG X, et al. Time Series Data Augmentation for Deep Learning: A Survey [C]//International Joint Conference on Artificial Intelligence. 2020.
[33] POPEL M, BOJAR O. Training Tips for the Transformer Model[J]. The Prague Bulletin of Mathematical Linguistics, 2018, 110: 43 ­ 70.
[34] CHUNG J, GULCEHRE C, CHO K, et al. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling[A]. 2014. arXiv: 1412.3555.
[35] JADERBERG M, SIMONYAN K, ZISSERMAN A, et al. Spatial transformer networks[J]. Advances in neural information processing systems, 2015, 28.
[36] BAHDANAU D, CHO K H, BENGIO Y. Neural machine translation by jointly learning to align and translate[C]//3rd International Conference on Learning Representations, ICLR 2015. 2015.
[37] KENTON J D M W C, TOUTANOVA L K. BERT: Pre­training of Deep Bidirectional Trans­ formers for Language Understanding[C]//Proceedings of NAACL­HLT. 2019: 4171­4186.
[38] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale[C]//International Conference on Learning Repre­ sentations. 2021.
[39] ARNAB A, DEHGHANI M, HEIGOLD G, et al. Vivit: A video vision transformer[C]// Proceedings of the IEEE/CVF international conference on computer vision. 2021: 6836­6846.
[40] GULATI A, QIN J, CHIU C C, et al. Conformer: Convolution­augmented Transformer for Speech Recognition[A]. 2020. arXiv: 2005.08100.
[41] LAMB A M, ALIAS PARTH GOYAL A G, ZHANG Y, et al. Professor forcing: A new algo­ rithm for training recurrent networks[J]. Advances in neural information processing systems, 2016, 29.
[42] KIPF T N, WELLING M. Semi­Supervised Classification with Graph Convolutional Networks [C]//International Conference on Learning Representations. 2017.
[43] GILMER J, SCHOENHOLZ S S, RILEY P F, et al. Neural message passing for quantum chemistry[C]//International conference on machine learning. PMLR, 2017: 1263­1272.
[44] DEFFERRARD M, BRESSON X, VANDERGHEYNST P. Convolutional neural networks on graphs with fast localized spectral filtering[J]. Advances in neural information processing systems, 2016, 29.
[45] VELIČKOVIĆ P, CUCURULL G, CASANOVA A, et al. Graph Attention Networks[C]// International Conference on Learning Representations. 2018.
[46] ZHU Y, XU W, ZHANG J, et al. A Survey on Graph Structure Learning: Progress and Oppor­ tunities[A]. 2022. arXiv: 2103.03036.
[47] LI R, WANG S, ZHU F, et al. Adaptive graph convolutional neural networks[C]//Proceedings of the AAAI conference on artificial intelligence: volume 32. 2018.
[48] ZHU Y, XU Y, YU F, et al. CAGNN: Cluster­Aware Graph Neural Networks for Unsupervised Graph Representation Learning[A]. 2020. arXiv: 2009.01674.
[49] CHEN Y, WU L, ZAKI M. Iterative deep graph learning for graph neural networks: Better and robust node embeddings[J]. Advances in neural information processing systems, 2020, 33: 19314­19326.
[50] 张昕, 曾鹏, 张瑞, 等. 交通大数据的特征及价值[J]. 软件导刊, 2016, 15(3): 3.
[51] 李德仁. 论时空大数据的智能处理与服务[J]. 地球信息科学学报, 2019, 21(12): 7.
[52] KIM T, KIM J, TAE Y, et al. Reversible instance normalization for accurate time­series forecasting against distribution shift[C]//International Conference on Learning Representations. 2021.
[53] JANG E, GU S, POOLE B. Categorical Reparameterization with Gumbel­Softmax[C]// International Conference on Learning Representations. 2017.
[54] ABU­EL­HAIJA S, PEROZZI B, KAPOOR A, et al. Mixhop: Higher­order graph convo­ lutional architectures via sparsified neighborhood mixing[C]//international conference on ma­ chine learning. PMLR, 2019: 21­29.
[55] KINGMA D P, BA J. Adam: A Method for Stochastic Optimization[C]//BENGIO Y, LECUN Y. 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7­9, 2015, Conference Track Proceedings. 2015.
[56] GUO S, LIN Y, FENG N, et al. Attention based spatial­temporal graph convolutional networks for traffic flow forecasting[C]//Proceedings of the AAAI conference on artificial intelligence: volume 33. 2019: 922­929.
[57] SONG C, LIN Y, GUO S, et al. Spatial­temporal synchronous graph convolutional networks: A new framework for spatial­temporal network data forecasting[C]//Proceedings of the AAAI conference on artificial intelligence: volume 34. 2020: 914­921.
[58] LI M, ZHU Z. Spatial­temporal fusion graph neural networks for traffic flow forecasting[C]// Proceedings of the AAAI conference on artificial intelligence: volume 35. 2021: 4189­4196.
[59] WANG D, CUI P, ZHU W. Structural deep network embedding[C]//Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining. 2016: 1225­ 1234.