TRAFFIC FLOW PREDICTION BASED ON PRE-TRAINED ENHANCED SPATIAL-TEMPORAL ATTENTION CONVOLUTIONAL NEURAL NETWORK

In a setting of swift economic growth, there are more and more different models on urban traffic roads every year, and the frequency of traffic accidents is rising. More precise and efficient traffic status prediction techniques are desperately needed.

Recent advancements in Machine Learning and Computer Vision have enriched our understanding of traffic prediction, yet unresolved problems persist. Firstly, the length of the input time series often has a linear or quadratic relationship with the computational complexity. However, existing models struggle with the computational complexity of long input time series, relying on a limited historical window to make predictions, thus impacting their ability to learn from long-term information explicitly. Secondly, spatial correlation of road networks varies dynamically over time, for example, the spatial correlation of traffic flows between office buildings and residential areas is very strong in the morning on weekdays but weak on weekends. However, existing studies usually use GCN to capture the spatial static correlation based on historical traffic flow without considering the spatial dynamic correlation.

To address the above two traffic prediction problems, our novel approach involves augmenting the downstream GCN with a versatile time series pre-trained model by utilizing Transformer and Attention Mechanism, named as Pre-trained Enhanced Spatial-Temporal Attention Convolutional Neural Network.

The proposed framework incorporates a scalable time series pre-trained model to improve the model's ability to learn from long-term historical data. Then, we introduce a spatial-temporal attention convolutional neural network (STACNN) to capture spatial dynamic correlation and extract temporal dynamic correlation with fewer layers and parameters.

On the one hand, the proposed pre-trained model, which is built using Transformer blocks, efficiently extracts contextually rich segment-level representations by learning temporal patterns from extensive historical time series. Because of its effective design, Transformer can be trained on one GPU and learn from very long-term sequences, such as weekly data. On the other hand, within the attention mechanism, the decoder can dynamically focus on different input sequence segments according to their significance for the current decoding step. As such, the proposed model can easily manage long input sequences and capture the dependencies between different components of the input and output sequence.

The proposed framework shows promise in enhancing the accuracy of traffic flow prediction by addressing key challenges in existing models, and thereby raise people's satisfaction index and traffic efficiency.

[1] MA T, ANTONIOU C, TOLEDO T. Hybrid machine learning algorithm and statistical time series model for network-wide traffic forecast[J]. Transportation Research Part C Emerging Technologies, 2020, 111(2020): 352-372.
[2] Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chengqi Zhang. 2019. Graph WaveNet for Deep Spatial-Temporal Graph Modeling. In IJCAI.
[3] TEDJOPURNOMO D A, BAO Z F, ZHENG B H, et al. A survey on modern deep neural network for traffic prediction: trends, methods and challenges[J]. IEEE Transactions on Knowledge and Data Engineering, 2022, 34(4): 1544-1561.
[4] LIU Jing, GUAN Wei. A summary of traffic flow forecasting methods[J]. Journal of Highway Transportation Research Development, 2004, 21(3): 82-85.
[5] MA X, DAI Z, HE Z, et al. Learning traffic as images: A deep convolutional neural network for large-scale transportation network speed prediction[J]. Sensors (Basel, Switzerland), 2017, 17 (4): 818.
[6] KAMARIANAKIS Y, GAO H O, PRASTACOS P. Characterizing regimes in daily cycles of urban traffic using smooth-transition regressions[J]. Transportation Research Part C Emerg-ing Technologies, 2010, 18(5): 821-840.
[7] DIA H. An object-oriented neural network approach to short-term traffic forecasting[J]. Eur. J. Oper. Res., 2001, 131(2): 253-261.
[8] ZHENG Z, SU D. Short-term traffic volume forecasting: A k-nearest neighbor approach enhanced by constrained linearly sewing principal component algorithm[J]. Transportation Re- search Part C Emerging Technologies, 2014, 43: 143-157.
[9] SMITH B L, WILLIAMS B M, OSWALD R K. Comparison of parametric and nonparametric models for traffic flow forecasting[J]. Transportation Research Part C, 2002, 10(4): 303-321.
[10] LIN X, HUANG Y. Short-term high-speed traffic flow prediction based on arima-garch-m model [J]. Wireless Personal Communications, 2021, 117(4): 3421-3430.
[11] CHEN C, HU J, MENG Q, et al. Short-time traffic flow prediction with arima-garch model[C]// 2011 IEEE Intelligent Vehicles Symposium (IV). IEEE, 2011: 607-612.
[12] FUSCO G, COLOMBARONI C, ISAENKO N. Short-term speed predictions exploiting big data on large urban road networks[J]. Transportation Research Part C Emerging Technologies, 2016, 73(12): 183-201.
[13] WILLIAMS B M, HOEL L A. Modeling and forecasting vehicular traffic flow as a seasonal arima process: Theoretical basis and empirical results[J]. Journal of Transportation Engi-neering, 2003, 129(6): 664-672.
[14] GUO J, HUANG W, WILLIAMS B M. Adaptive kalman filter approach for stochastic short-term traffic flow rate prediction and uncertainty quantification[J]. Transportation Research Part C: Emerging Technologies, 2014, 43: 50-64.
[15] XU D W, WANG Y D, JIA L M, et al. Real-time road traffic state prediction based on arima and kalman filter[J]. Frontiers of Information Technology & Electronic Engineering, 2017, 18 (2): 287-302.
[16] TIAN B, WANG G, XU Z, et al. Communication delay compensation for string stability of caccsystem using lstm prediction[J]. Vehicular Communications, 2021, 29: 100333. 59REFERENCES
[17] ABDOOS M, BAZZAN A. Hierarchical traffic signal optimization using reinforcement learning and traffic prediction with long-short term memory[J]. Expert Systems with Appli-cations, 2021, 171: 114580.
[18] ZHANG H, SONG C, ZHANG J, et al. A multi-step airport delay prediction model based on spatial-temporal correlation and auxiliary features[J]. IET Intelligent Transport Systems, 2021, 15(7): 916-928.
[19] LIU H, TIAN H Q, LI Y F. Comparison of two new arima-ann and arima-kalman hybrid methods for wind speed prediction[J]. Applied Energy, 2012, 98: 415-424.
[20] LI W, CHEN S, WANG X, et al. A hybrid approach for short-term traffic flow forecasting based on similarity identification[J]. Modern Physics Letters B, 2021, 35(13): 2150212.
[21] ZHAO L, SONG Y, ZHANG C, et al. T-gcn: A temporal graph convolutional network for traffic prediction[J]. IEEE Transactions on Intelligent Transportation Systems, 2019, PP(99): 1-11.
[22] GUO S, LIN Y, FENG N, et al. Attention based spatial-temporal graph convolutional networks for traffic flow forecasting[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2019, 33(01): 922-929.
[23] LI Y, YU R, SHAHABI C, et al. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting[J]. arXiv: Learning, 2018.
[24] GE L, LI S, WANG Y, et al. Global spatial-temporal graph convolutional network for urban traffic speed prediction[J]. Applied Sciences, 2020, 10(4): 1509.
[25] YU B, YIN H, ZHU Z. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting[J]. arXiv preprint arXiv:1709.04875, 2017.
[26] SUN S, WU H, XIANG L. City-wide traffic flow forecasting using a deep convolutional neural network[J]. Sensors, 2020, 20(2): 421.
[27] TAO L, GU Y, LU W, et al. An attention-based approach for traffic conditions forecasting considering spatial-temporal features[J]. 2020 IEEE 5th International Conference on Intelli-gent Transportation Engineering (ICITE), 2020: 117-122.
[28] CHEN W, CHEN L, XIE Y, et al. Multi-range attentive bicomponent graph convolutional network for traffic forecasting[J]. Proceedings of the AAAI Conference on Artificial Intelli-gence, 2020, 34(4): 3529-3536.
[29] SONG C, LIN Y, GUO S, et al. Spatial-temporal synchronous graph convolutional networks: A new framework for spatial-temporal network data forecasting[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(1): 914-921.
[30] REN S, HAN L, LI Z, et al. Spatial-temporal traffic speed bands data analysis and prediction[J]. 2017 IEEE International Conference on Industrial Engineering and Engineering Management (IEEM), 2017: 808-812.
[31] LIU Y, ZHENG H, FENG X, et al. Short-term traffic flow prediction with conv-lstm[J]. 2017 9th International Conference on Wireless Communications and Signal Processing (WCSP), 2017: 1-6.
[32] FENG D, WU Z, ZHANG J, et al. Dynamic global-local spatial-temporal network for traffic speed prediction[J]. IEEE Access, 2020, 8: 209296-209307. 60REFERENCES
[33] YAO H, TANG X, WEI H, et al. Revisiting spatial-temporal similarity: A deep learning framework for traffic prediction[C]//Proceedings of the AAAI conference on artificial intelli-gence: volume 33. 2019: 5668-5675.
[34] DONG C, FU SHAO C, RICHARDS S H, et al. Flow rate and time mean speed predictions for the urban freeway network using state space models[J]. Transportation Research Part C-emerging Technologies, 2014, 43: 20-32.
[35] MA T, ZHOU Z, ANTONIOU C. Dynamic factor model for network traffic state forecast[J]. Transportation Research Part B: Methodological, 2018, 118: 281-317.
[36] ZHENG L, YANG J, CHEN L, et al. Dynamic spatial-temporal feature optimization with eri big data for short-term traffic flow prediction[J]. Neurocomputing, 2020, 412: 339-350.
[37] SHI X J, CHEN Z R, WANG H, et al. Convolutional LSTM network: a machine learning approach for precipitation nowcasting[C]//29th Annual Conference on Neural Information Processing Systems. Montreal: NIPS, 2015: 802-810.
[38] YAO H X, TANG X F, WEI H, et al. Revisiting spatial-temporal similarity: a deep learning framework for traffic prediction[C]//Proceedings of the Thirty- Third AAAI Conference on Artificial Intelligence. Honolulu: AAAI, 2019: 5668-5675.
[39] ZHANG J, ZHENG Y, QI D. Deep spatiotemporal residual networks for citywide crowd flows prediction[C]//Proceedings of the Thirty-First AAAI Conference on Artificial Intelli-gence. San Francisco: AAAI, 2016: 1655-1661.
[40] ZHAO L, SONG Y J, ZHANG C, et al. T-GCN: a temporal graph convolutional network for traffic prediction[J]. IEEE Transactions on Intelligent Transportation Systems， 2020, 21(9): 3848-3858.
[41] YU B, YIN H T, ZHU Z X. Spatio-temporal graph convolutional networks: a deep learning framework for traffic forecasting[C]//Proceedings of the Twenty- Seventh International Joint Conference on Artificial Intelligence. Stockholm: IJCAI, 2018: 3634-3640.
[42] Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. 2018. Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting. In ICLR.
[43] Chao Shang, Jie Chen, and Jinbo Bi. 2021. Discrete Graph Structure Learning for Forecasting Multiple Time Series. In ICLR.
[44] Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. 2020. Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Net-works. In SIGKDD.
[45] Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chengqi Zhang. 2019. Graph WaveNet for Deep Spatial-Temporal Graph Modeling. In IJCAI.
[46] Chuanpan Zheng, Xiaoliang Fan, Cheng Wang, and Jianzhong Qi. 2020. GMAN: A Graph Multi-Attention Network for Traffic Prediction. In AAAI.
[47] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language Models are Few-Shot Learners. In NeurIPS.
[48] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL.
[49] Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL.
[50] George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. 2021. A Transformer-based Framework for Multivariate Time Series Represen-tation Learning. In SIGKDD.
[51] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xi- aohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image is Worth 16x16 Words: Trans-formers for Image Recognition at Scale. In ICLR.
[52] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In NeurIPS.
[53] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2021. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377 (2021).
[54] VELIKOVI P, CUCURULL G, CASANOVA A, et al. Graph attention networks[C]//6th International Conference on Learning Representations. Vancouver: ICLR, 2018: 1-12.
[55] FENG X C, GUO J, QIN B, et al. Effective deep memory networks for distant supervised relation extraction[C]//Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence.
[56] Melbourne: IJCAI, 2017: 4002-4008.KIPF T N, WELLING M. Semi-supervised classification with graph convolutional networks [C]//5th International Conference on Learning Representations. Toulon: ICLR, 2017: 1-14.
[57] SIMONOVSKY M, KOMODAKIS N. Dynamic edge-conditioned filters in convolutional neural networks on graphs[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 29-38.
[58] Chao Chen, Karl Petty, Alexander Skabardonis, Pravin Varaiya, and Zhanfeng Jia. 2001. Freeway performance measurement system: mining loop detector data. Transportation Research Record (2001).
[59] Shengnan Guo, Youfang Lin, Ning Feng, Chao Song, and Huaiyu Wan. 2019. Attention Based Spatial-Temporal Graph Convolutional Networks for Traffic Flow Forecasting. In AAAI.
[60] Hosagrahar V Jagadish, Johannes Gehrke, Alexandros Labrinidis, Yannis Papakonstan-tinou, Jignesh M Patel, Raghu Ramakrishnan, and Cyrus Shahabi. 2014. Big data and its technical challenges. Commun. ACM (2014).
[61] Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chengqi Zhang. 2019. Graph WaveNet for Deep Spatial-Temporal Graph Modeling. In IJCAI.
[62] Shengnan Guo, Youfang Lin, Huaiyu Wan, Xiucheng Li, and Gao Cong. 2021. Learning dynamics and heterogeneity of spatial-temporal graph data for traffic forecasting. TKDE (2021).
[63] Zheng Lu, Chen Zhou, Jing Wu, Hao Jiang, and Songyue Cui. 2016. Integrating Granger Causality and Vector Auto-Regression for Traffic Prediction of Large-Scale WLANs. KSII Trans. Internet Inf. Syst. (2016).
[64] Alexander J. Smola and Bernhard Schölkopf. 2004. A tutorial on support vector regression. Stat. Comput. (2004).
[65] Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to Sequence Learning with Neural Networks. In NeurIPS.
[66] Chao Song, Youfang Lin, Shengnan Guo, and Huaiyu Wan. 2020. Spatial-Temporal Synchronous Graph Convolutional Networks: A New Framework for Spatial-Temporal Network Data Forecasting. In AAAI.
[67] YU B, YIN H T, ZHU Z X. Spatio-temporal graph convolutional networks: a deep learning framework for traffic forecasting[C]//Proceedings of the Twenty- Seventh International Joint Conference on Artificial In- telligence. Stockholm: IJCAI, 2018: 3634-3640.
[68] Sami Abu-El-Haija, Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard, Kristina Lerman, Hrayr Harutyunyan, Greg Ver Steeg, and Aram Galstyan. 2019. MixHop: Higher-Order Graph Convolutional Architectures via Sparsified Neighborhood Mixing. In ICML.
[69] Yongjun Xu, Xin Liu, Xin Cao, Changping Huang, Enke Liu, Sen Qian, Xingchen Liu, Yanjun Wu, Fengliang Dong, Cheng-Wei Qiu, et al. 2021. Artificial intelligence: A powerful paradigm for scientific research. The Innovation 2, 4 (2021).
[70] Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering. In NeurIPS.
[71] Thomas N. Kipf and Max Welling. 2017. Semi-Supervised Classification with Graph Convolutional Networks. In ICLR.
[72] Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the Properties of Neural Machine Translation: Encoder-Decoder Approaches. In SSST@EMNLP.
[73] Bing Yu, Haoteng Yin, and Zhanxing Zhu. 2018. Spatio-Temporal Graph Con- volutional Networks: A Deep Learning Framework for Traffic Forecasting. In IJCAI.
[74] Defu Cao, Yujing Wang, Juanyong Duan, Ce Zhang, Xia Zhu, Congrui Huang, Yunhai Tong, Bixiong Xu, Jing Bai, Jie Tong, and Qi Zhang. 2020. Spectral Temporal Graph Neural Network for Multivariate Time-series Forecasting. In NeurIPS.
[75] Zheyi Pan, Yuxuan Liang, Weifeng Wang, YongYu, Yu Zheng, and Junbo Zhang. 2019. Urban traffic prediction from spatio-temporal data using deep meta learning. In SIGKDD. 1720–1730.
[76] Lei Bai, Lina Yao, Can Li, Xianzhi Wang, and Can Wang. 2020. Adaptive Graph Convolutional Recurrent Network for Traffic Forecasting. In NeurIPS.
[77] Ling Zhao, Yujiao Song, Chao Zhang, Yu Liu, Pu Wang, Tao Lin, Min Deng, and Haifeng Li. 2020. T-GCN: A Temporal Graph Convolutional Network for Traffic Prediction. IEEE TITS (2020).
[78] Luca Franceschi, Mathias Niepert, Massimiliano Pontil, and Xiao He. 2019. Learn- ing Discrete Structures for Graph Neural Networks. In ICML.
[79] Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. 2018. Neural relational inference for interacting systems. In ICML.