Explainable Cognitive States Decoding Based on fMRI via Graph Neural Networks

Cognitive state decoding also referred to simply as brain decoding, plays a crucial role in advancing the field of neuroscience and facilitating the treatment of brain disorders. By decoding neural signals into corresponding brain states, such as motor, emotional processing, and decision-making states, cognitive state decoding has significant implications for understanding the workings of various brain functions. With the rapid development of deep learning, data-driven brain decoding methods have emerged as an advantageous approach for end-to-end learning and processing high-dimensional data. While neural network architectures with deeper layers and larger parameter sizes have made significant breakthroughs in Euclidean space data, deep learning-based brain decoding models still face limitations in decoding accuracy and model explainability for non-Euclidean neural data. The brain processes information through the collaboration of different brain regions and the integration of temporal information at various temporal and spatial scales to effectively perceive and understand the external world. 

Inspired by the information processing and time integration mechanisms of the brain, this thesis proposes a novel, brain-inspired, geometric deep learning-based neural network architecture, the Spatial Temporal-pyramid Graph Convolutional Network (STpGCN). The STpGCN is designed with a multi-scale spatial-temporal pyramid structure and a bottom-up pathway. By simulating the brain's information processing and time integration mechanisms, the STpGCN explicitly utilizes the time-dependency of multi-scale brain activity to achieve state-of-the-art performance in brain decoding of functional magnetic resonance imaging (fMRI) data across 23 cognitive tasks in the Human Connectome Project S1200. In addition, to address the issue of poor model explainability in deep learning, this thesis proposes a post-hoc model-agnostic sensitivity analysis method, BrainNetX. By perturbing and analyzing the full-brain-scale network, BrainNetX identifies and annotates the brain networks that significantly affect the model decoding process for cognitive tasks. Without requiring model retraining, BrainNetX can perform sensitivity analysis on brain network features of pre-trained machine learning models on different tasks, achieving the goal of annotating task-related brain networks. Through explainability analysis of the STpGCN, BrainNetX effectively annotates the key brain regions related to cognitive tasks. This not only demonstrates that the STpGCN has excellent explainability but also indicates that BrainNetX can be effectively used for model explainability analysis.

In summary, this thesis proposes the STpGCN model and BrainNetX sensitivity analysis method. Through experimental analysis on a large-scale fMRI dataset, the STpGCN outperforms the existing best models in brain decoding across 23 cognitive tasks. Furthermore, the adoption of the BrainNetX sensitivity analysis method can effectively annotate key brain regions related to cognitive tasks. Based on the post-hoc analysis of these regions, the hierarchical structure design of the STpGCN is found to significantly improve the model's explainability, robustness, and generalization ability. This thesis not only provides a feasible solution for representation learning of the brain activity in various cognitive tasks but also demonstrates the great potential of brain-inspired and geometric deep learning approaches for completing brain decoding tasks. By combining the STpGCN and BrainNetX, this thesis provides a research framework for brain decoding in the field of neuroscience with higher decoding performance and better explainability.

大脑认知功能解码在神经科学领域具有举足轻重的地位，为推动脑科学发展和脑疾病治疗提供了关键支持。大脑认知功能解码是将不同神经信号解码为相应的认知任务，如运动、情感和决策等。这对于研究和理解大脑在不同任务具有重要意义。随着深度学习领域的快速发展，基于数据驱动的神经解码方法在端对端特征学习和处理高维数据方面具有明显优势。层数更深、参数规模更大的神经网络架构已在欧氏空间数据上取得了显著突破。然而，针对非欧空间的神经数据，基于深度学习的神经解码模型在解码准确率以及模型可解释性方面仍存在局限性。

在处理信息时，大脑通过各个脑区的相互协作与时间信息整合，在不同的时间和空间尺度上处理信息，进而有效地感知和理解外部世界。受大脑处理信息的方式启发，本文提出一种基于几何深度学习的神经网络架构，时空金字塔图卷积网络 (Spatial Temporal-pyramid Graph Convolutional Network, STpGCN)。时空金字塔图卷积网络模拟大脑的层级化结构，在设计上包含多尺度时空通路和自下而上的信息聚合通路。通过模拟大脑中信息处理和时间整合方式，时空金字塔图卷积网络能够显式地利用多尺度大脑活动的时间依赖性，实现在人类连接组计划 (HCP) S1200的23项认知任务下对功能磁共振成像数据进行脑解码的当前最佳性能。
此外，为改善深度学习模型可解释性差的问题，本文提出了一种与模型无关的事后敏感性分析方法：BrainNetX。该方法通过扰动的方式从全脑尺度分析并标注模型解码过程中对认知任务解码有显著影响的脑网络。BrainNetX在不需要重新训练模型的前提下，能在不同任务上对已经训练好的机器学习模型进行在脑网络层面的显著性分析，从而实现标注与任务相关的脑网络的目的。通过对时空金字塔图卷积网络进行可解释性分析，BrainNetX有效标注出与任务相关的脑区。该结果不仅证明时空金字塔图卷积网络具备良好的可解释性，还表明BrainNetX可有效用于分析模型的可解释性。

综上所述，本文提出了时空金字塔图卷积网络（STpGCN）与BrainNetX敏感性分析方法。通过在大规模功能磁共振成像数据集的实验分析，时空金字塔图卷积网络在对23项认知任务的神经解码中取得了优于现有最佳模型的性能表现。同时，采用BrainNetX敏感性分析方法能够有效标注出与23项认知任务相关的脑部区域。基于这些区域的事后分析，进一步印证了时空金字塔图卷积网络中的层次结构设计对于提高模型的可解释性、鲁棒性和泛化能力具有显著作用。本文不仅为大脑在多种认知任务下的信息表征学习提供了一种切实可行的方案，也展示出脑启发与几何深度学习方式在完成大脑认知状态解码任务方面的巨大潜力。通过将时空金字塔图卷积网络和BrainNetX敏感性分析方法相结合，本文为神经科学领域提供了具有更高解码性能与更好可解释性的神经解码研究框架。

[1] FRISTON K J. Functional and effective connectivity: a review[J]. Brain connectivity, 2011, 1 (1): 13-36.
[2] PARK H J, FRISTON K. Structural and functional brain networks: from connections to cognition[J]. Science, 2013, 342(6158): 1238411.
[3] DECO G, TONONI G, BOLY M, et al. Rethinking segregation and integration: contributions of whole-brain modelling[J]. Nature reviews neuroscience, 2015, 16(7): 430-439.
[4] DECO G, KRINGELBACH M L. Metastability and coherence: extending the communication through coherence hypothesis using a whole-brain computational perspective[J]. Trends in neurosciences, 2016, 39(3): 125-135.
[5] HAYNES J D. A primer on pattern-based approaches to fMRI: principles, pitfalls, and perspectives[J]. Neuron, 2015, 87(2): 257-270.
[6] ITO T, HEARNE L, MILL R, et al. Discovering the computational relevance of brain network organization[J]. Trends in cognitive sciences, 2020, 24(1): 25-38.
[7] KAPOGIANNIS D, MATTSON M P. Disrupted energy metabolism and neuronal circuit dysfunction in cognitive impairment and Alzheimer’s disease[J]. The lancet neurology, 2011, 10 (2): 187-198.
[8] LIVINGSTON G, HUNTLEY J, SOMMERLAD A, et al. Dementia prevention, intervention, and care: 2020 report of the Lancet Commission[J]. The lancet, 2020, 396(10248): 413-446.
[9] MESULAM M M. From sensation to cognition.[J]. Brain: a journal of neurology, 1998, 121 (6): 1013-1052.
[10] POLDRACK R A. Can cognitive processes be inferred from neuroimaging data?[J]. Trends in cognitive sciences, 2006, 10(2): 59-63.
[11] BRESSLER S L, MENON V. Large-scale brain networks in cognition: emerging methods and principles[J]. Trends in cognitive sciences, 2010, 14(6): 277-290.
[12] KRIEGESKORTE N, DOUGLAS P K. Cognitive computational neuroscience[J]. Nature neuroscience, 2018, 21(9): 1148-1160.
[13] WHITTINGTON J C, MCCAFFARY D, BAKERMANS J J, et al. How to build a cognitive map[J]. Nature neuroscience, 2022, 25(10): 1257-1272.
[14] DEHAENE S, LE CLEC’H G, COHEN L, et al. Inferring behavior from functional brain images [J]. Nature neuroscience, 1998, 1(7): 549-549.
[15] GEERLIGS L, RUBINOV M, HENSON R N, et al. State and trait components of functional connectivity: individual differences vary with mental state[J]. Journal of neuroscience, 2015, 35(41): 13949-13961.
[16] SMITH S M, NICHOLS T E. Statistical challenges in “big data” human neuroimaging[J]. Neuron, 2018, 97(2): 263-268.
[17] DROBYSHEVSKY A, BAUMANN S B, SCHNEIDER W. A rapid fMRI task battery for mapping of visual, motor, cognitive, and emotional function[J]. Neuroimage, 2006, 31(2): 732-744.
[18] MILLER M B, DONOVAN C L, VAN HORN J D, et al. Unique and persistent individual patterns of brain activity across different memory retrieval tasks[J]. Neuroimage, 2009, 48(3): 625-635.
[19] WOO C W, CHANG L J, LINDQUIST M A, et al. Building better biomarkers: brain models in translational neuroimaging[J]. Nature neuroscience, 2017, 20(3): 365-377.
[20] LIVEZEY J A, GLASER J I. Deep learning approaches for neural decoding across architectures and recording modalities[J]. Briefings in bioinformatics, 2021, 22(2): 1577-1591.
[21] SCHWEMMER M A, SKOMROCK N D, SEDERBERG P B, et al. Meeting brain–computer interface user performance expectations using a deep neural network decoding framework[J]. Nature medicine, 2018, 24(11): 1669-1676.
[22] DU B, CHENG X, DUAN Y, et al. fMRI brain decoding and its applications in brain–computer interface: a survey[J]. Brain sciences, 2022, 12(2): 228.
[23] XU L, XU M, JUNG T P, et al. Review of brain encoding and decoding mechanisms for EEGbased brain–computer interface[J]. Cognitive neurodynamics, 2021, 15: 569-584.
[24] AMUNTS K, EBELL C, MULLER J, et al. The human brain project: creating a European research infrastructure to decode the human brain[J]. Neuron, 2016, 92(3): 574-581.
[25] MA Y, GONG A, NAN W, et al. Personalized Brain–Computer Interface and Its Applications [J]. Journal of personalized medicine, 2023, 13(1): 46.
[26] GROOTSWAGERS T, WARDLE S G, CARLSON T A. Decoding dynamic brain patterns from evoked responses: A tutorial on multivariate pattern analysis applied to time series neuroimaging data[J]. Journal of cognitive neuroscience, 2017, 29(4): 677-697.
[27] RITCHIE J B, KAPLAN D M, KLEIN C. Decoding the brain: Neural representation and the limits of multivariate pattern analysis in cognitive neuroscience[J]. The British journal for the philosophy of science, 2019.
[28] NORMAN K A, POLYN S M, DETRE G J, et al. Beyond mind-reading: multi-voxel pattern analysis of fMRI data[J]. Trends in cognitive sciences, 2006, 10(9): 424-430.
[29] CARLSON T A, SCHRATER P, HE S. Patterns of activity in the categorical representations of objects[J]. Journal of cognitive neuroscience, 2003, 15(5): 704-717.
[30] MOURAO-MIRANDA J, BOKDE A L, BORN C, et al. Classifying brain states and determining the discriminating activation patterns: support vector machine on functional MRI data[J]. NeuroImage, 2005, 28(4): 980-995.
[31] HAXBY J V, GOBBINI M I, FUREY M L, et al. Distributed and overlapping representations of faces and objects in ventral temporal cortex[J]. Science, 2001, 293(5539): 2425-2430.
[32] KRIEGESKORTE N, GOEBEL R, BANDETTINI P. Information-based functional brain mapping[J]. Proceedings of the national academy of sciences, 2006, 103(10): 3863-3868.
[33] PEREIRA F, MITCHELL T, BOTVINICK M. Machine learning classifiers and fMRI: a tutorial overview[J]. Neuroimage, 2009, 45(1): S199-S209.
[34] COX D D, SAVOY R L. Functional magnetic resonance imaging (fMRI)“brain reading”: detecting and classifying distributed patterns of fMRI activity in human visual cortex[J]. Neuroimage, 2003, 19(2): 261-270.
[35] CONROY B R, SINGER B D, GUNTUPALLI J S, et al. Inter-subject alignment of human cortical anatomy using functional connectivity[J]. NeuroImage, 2013, 81: 400-411.
[36] HAXBY J V, GUNTUPALLI J S, CONNOLLY A C, et al. A common, high-dimensional model of the representational space in human ventral temporal cortex[J]. Neuron, 2011, 72(2): 404- 416.
[37] KOYAMADA S, SHIKAUCHI Y, NAKAE K, et al. Deep learning of fMRI big data: a novel approach to subject-transfer decoding[A]. 2015.
[38] Li, Hongming and Fan, Yong. Brain decoding from functional MRI using long short-term memory recurrent neural networks[C]//International conference on medical image computing and computer-assisted intervention. Springer, 2018: 320-328.
[39] LI H, FAN Y. Interpretable, highly accurate brain decoding of subtly distinct brain states from functional MRI using intrinsic functional networks and long short-term memory recurrent neural networks[J]. NeuroImage, 2019, 202: 116059.
[40] GAO Y, ZHANG Y, CAO Z, et al. Decoding brain states from fMRI signals by using unsupervised domain adaptation[J]. IEEE journal of biomedical and health informatics, 2019, 24(6): 1677-1685.
[41] WANG X, LIANG X, JIANG Z, et al. Decoding and mapping task states of the human brain via deep learning[J]. Human brain mapping, 2020, 41(6): 1505-1519.
[42] ZHANG Y, TETREL L, THIRION B, et al. Functional annotation of human cognitive states using deep graph convolution[J]. NeuroImage, 2021, 231: 117847.
[43] SATTERTHWAITE T D, ELLIOTT M A, RUPAREL K, et al. Neuroimaging of the Philadelphia neurodevelopmental cohort[J]. Neuroimage, 2014, 86: 544-553.
[44] JACK JR C R, BERNSTEIN M A, FOX N C, et al. The Alzheimer’s disease neuroimaging initiative (ADNI): MRI methods[J]. Journal of magnetic resonance imaging, 2008, 27(4): 685- 691.
[45] DI MARTINO A, YAN C G, LI Q, et al. The autism brain imaging data exchange: towards a large-scale evaluation of the intrinsic brain architecture in autism[J]. Molecular psychiatry, 2014, 19(6): 659-667.
[46] VAN ESSEN D C, SMITH S M, BARCH D M, et al. The WU-Minn human connectome project: an overview[J]. Neuroimage, 2013, 80: 62-79.
[47] ZHOU J, CUI G, HU S, et al. Graph neural networks: A review of methods and applications [J]. AI open, 2020, 1: 57-81.
[48] KIPF T N, WELLING M. Semi-supervised classification with graph convolutional networks [C]//International conference on learning representations. 2017.
[49] VELIČKOVIĆ P, CUCURULL G, CASANOVA A, et al. Graph attention networks[C]// International conference on learning representations. 2018.
[50] XU K, HU W, LESKOVEC J, et al. How powerful are graph neural networks?[C]//International conference on learning representations. 2018.
[51] YU B, YIN H, ZHU Z. Spatio-temporal graph convolutional networks: a deep learning framework for traffic forecasting[C]//Proceedings of the international joint conference on artificial intelligence. 2018: 3634-3640.
[52] YAN S, XIONG Y, LIN D. Spatial temporal graph convolutional networks for skeleton-based action recognition[C]//AAAI conference on artificial intelligence. 2018.
[53] CHENG K, ZHANG Y, HE X, et al. Skeleton-based action recognition with shift graph convolutional network[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 183-192.
[54] SHI L, ZHANG Y, CHENG J, et al. Skeleton-based action recognition with multi-stream adaptive graph convolutional networks[J]. IEEE Transactions on image processing, 2020, 29: 9532- 9545.
[55] SONG Y F, ZHANG Z, SHAN C, et al. Constructing stronger and faster baselines for skeletonbased action recognition[J]. IEEE Transactions on pattern analysis and machine intelligence, 2022, 45(2): 1474-1488.
[56] SONG C, LIN Y, GUO S, et al. Spatial-temporal synchronous graph convolutional networks: A new framework for spatial-temporal network data forecasting[C]//Proceedings of the AAAI conference on artificial intelligence: volume 34. 2020: 914-921.
[57] WU Z, PAN S, LONG G, et al. Connecting the dots: Multivariate time series forecasting with graph neural networks[C]//Proceedings of the ACM SIGKDD International conference on knowledge discovery and data mining. 2020: 753-763.
[58] HAN L, DU B, SUN L, et al. Dynamic and multi-faceted spatio-temporal deep learning for traffic speed forecasting[C]//Proceedings of the ACM SIGKDD conference on knowledge discovery and data mining. 2021: 547-555.
[59] SHAO W, PENG Y, ZU C, et al. Hypergraph based multi-task feature selection for multimodal classification of Alzheimer’s disease[J]. Computerized medical imaging and graphics, 2020, 80: 101663.
[60] LI X, ZHOU Y, DVORNEK N, et al. BrainGNN: Interpretable brain graph neural network for fmri analysis[J]. Medical image analysis, 2021, 74: 102233.
[61] ZU C, GAO Y, MUNSELL B, et al. Identifying disease-related subnetwork connectome biomarkers by sparse hypergraph learning[J]. Brain imaging and behavior, 2019, 13(4): 879- 892.
[62] LIU M, GAO Y, YAP P T, et al. Multi-hypergraph learning for incomplete multimodality data [J]. IEEE journal of biomedical and health informatics, 2017, 22(4): 1197-1208.
[63] CUI H, DAI W, ZHU Y, et al. BrainGB: a benchmark for brain network analysis with graph neural networks[J]. IEEE Transactions on medical imaging, 2022.
[64] BESSADOK A, MAHJOUB M A, REKIK I. Graph neural networks in network neuroscience [J]. IEEE Transactions on pattern analysis and machine intelligence, 2022.
[65] LI X, ZHOU Y, DVORNEK N C, et al. Pooling regularized graph neural network for fmri biomarker analysis[C]//International conference on medical image computing and computerassisted intervention. Springer, 2020: 625-635.
[66] XIAO L, WANG J, KASSANI P H, et al. Multi-hypergraph learning-based brain functional connectivity analysis in fMRI data[J]. IEEE Transactions on medical imaging, 2019, 39(5): 1746-1758.
[67] GADGIL S, ZHAO Q, PFEFFERBAUM A, et al. Spatio-temporal graph convolution for restingstate fmri analysis[C]//International conference on medical image computing and computerassisted intervention. Springer, 2020: 528-538.
[68] KIM B H, YE J C, KIM J J. Learning dynamic graph representation of brain connectome with spatio-temporal attention[C]//Advances in neural information processing systems: volume 34. 2021: 4314-4327.
[69] KAN X, DAI W, CUI H, et al. Brain network transformer[C]//Advances in neural information processing systems. 2022.
[70] LUNDBERG S M, LEE S I. A unified approach to interpreting model predictions[M]//Advances in neural information processing systems: volume 30. 2017.
[71] LUNDBERG S M, ERION G, CHEN H, et al. From local explanations to global understanding with explainable AI for trees[J]. Nature machine intelligence, 2020, 2(1): 56-67.
[72] COVERT I, LUNDBERG S M, LEE S I. Understanding global feature contributions with additive importance measures[C]//Advances in neural information processing systems: volume 33. 2020: 17212-17223.
[73] YUAN H, YU H, GUI S, et al. Explainability in graph neural networks: A taxonomic survey [J]. IEEE Transactions on pattern analysis and machine intelligence, 2022.
[74] GEVREY M, DIMOPOULOS I, LEK S. Review and comparison of methods to study the contribution of variables in artificial neural network models[J]. Ecological modelling, 2003, 160(3): 249-264.
[75] BALDASSARRE F, AZIZPOUR H. Explainability Techniques for Graph Convolutional Networks[C]//International conference on machine learning (ICML) workshops. 2019.
[76] SPRINGENBERG J, DOSOVITSKIY A, BROX T, et al. Striving for Simplicity: The All Convolutional Net[C]//International conference on learning representations (ICLR) workshop. 2015.
[77] SHRIKUMAR A, GREENSIDE P, KUNDAJE A. Learning important features through propagating activation differences[C]//International conference on machine learning. PMLR, 2017: 3145-3153.
[78] POPE P E, KOLOURI S, ROSTAMI M, et al. Explainability methods for graph convolutional neural networks[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019: 10772-10781.
[79] HUANG Q, YAMADA M, TIAN Y, et al. Graphlime: Local interpretable model explanations for graph neural networks[J]. IEEE Transactions on knowledge and data engineering, 2022: 1-6.
[80] VU M, THAI M T. Pgm-explainer: Probabilistic graphical model explanations for graph neural networks[C]//Advances in neural information processing systems: volume 33. 2020: 12225- 12235.
[81] ZHANG Y, DEFAZIO D, RAMESH A. Relex: A model-agnostic relational model explainer [C]//Proceedings of the 2021 AAAI/ACM Conference on AI, ethics, and society. 2021: 1042- 1049.
[82] SCHWARZENBERG R, HÜBNER M, HARBECKE D, et al. Layerwise relevance visualization in convolutional text graph classifiers[C]//Proceedings of the Workshop on Graph-Based Methods for Natural Language Processing. 2019: 58-62.
[83] FENG Q, LIU N, YANG F, et al. DEGREE: Decomposition based explanation for graph neural networks[C]//International conference on learning representations. 2022.
[84] YUAN H, TANG J, HU X, et al. XGNN: Towards model-level explanations of graph neural networks[C]//Proceedings of the ACM SIGKDD International conference on knowledge discovery and data mining. 2020: 430-438.
[85] SHAN C, SHEN Y, ZHANG Y, et al. Reinforcement learning enhanced explainer for graph neural networks[C]//Advances in neural information processing systems: volume 34. 2021: 22523-22533.
[86] COVERT I C, LUNDBERG S, LEE S I. Explaining by removing: A unified framework for model explanation[J]. The journal of machine learning research, 2021, 22(1): 9477-9566.
[87] GOODWIN N L, NILSSON S R, CHOONG J J, et al. Toward the explainability, transparency, and universality of machine learning for behavioral classification in neuroscience[J]. Current opinion in neurobiology, 2022, 73: 102544.
[88] YING Z, BOURGEOIS D, YOU J, et al. Gnnexplainer: Generating explanations for graph neural networks[M]//Advances in neural information processing systems: volume 32. 2019.
[89] LUO D, CHENG W, XU D, et al. Parameterized explainer for graph neural network[C]// Advances in neural information processing systems: volume 33. 2020: 19620-19631.
[90] FUNKE T, KHOSLA M, RATHEE M, et al. Zorro: Valid, sparse, and stable explanations in graph neural networks[J]. IEEE Transactions on knowledge and data engineering, 2022.
[91] WANG X, WU Y, ZHANG A, et al. Towards multi-grained explainability for graph neural networks[C]//Advances in neural information processing systems: volume 34. 2021: 18446- 18458.
[92] SCHLICHTKRULL M S, DE CAO N, TITOV I. Interpreting Graph Neural Networks for NLP With Differentiable Edge Masking[C]//International conference on learning representations. 2021.
[93] YUAN H, YU H, WANG J, et al. On explainability of graph neural networks via subgraph explorations[C]//International conference on machine learning. PMLR, 2021: 12241-12252.
[94] KINGMA D P, WELLING M. Auto-encoding variational bayes[J]. CoRR, 2013, abs/1312.6114.
[95] DABKOWSKI P, GAL Y. Real time image saliency for black box classifiers[M]//Advances in neural information processing systems: volume 30. 2017.
[96] KOCSIS L, SZEPESVÁRI C. Bandit based monte-carlo planning[C]//European conference on machine learning. Springer, 2006: 282-293.
[97] ŠTRUMBELJ E, KONONENKO I. Explaining prediction models and individual predictions with feature contributions[J]. Knowledge and information systems, 2014, 41: 647-665.
[98] SHAPLEY L S, et al. A value for n-person games[M]. Princeton University Press Princeton, 1953.
[99] BULLMORE E, SPORNS O. Complex brain networks: graph theoretical analysis of structural and functional systems[J]. Nature reviews neuroscience, 2009, 10(3): 186-198.
[100] Bullmore, Ed and Sporns, Olaf. The economy of brain network organization[J]. Nature reviews neuroscience, 2012, 13(5): 336-349.
[101] BUCKNER R L, DINICOLA L M. The brain’s default network: updated anatomy, physiology and evolving insights[J]. Nature reviews neuroscience, 2019, 20(10): 593-608.
[102] DECO G, VIDAURRE D, KRINGELBACH M L. Revisiting the global workspace orchestrating the hierarchical organization of the human brain[J]. Nature human behaviour, 2021, 5(4): 497- 511.
[103] CAUCHETEUX C, GRAMFORT A, KING J R. Evidence of a predictive coding hierarchy in the human brain listening to speech[J]. Nature human behaviour, 2023: 1-12.
[104] DING N, MELLONI L, ZHANG H, et al. Cortical tracking of hierarchical linguistic structures in connected speech[J]. Nature neuroscience, 2016, 19(1): 158-164.
[105] LIN T Y, DOLLÁR P, GIRSHICK R, et al. Feature pyramid networks for object detection [C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 2117-2125.
[106] YARKONI T, POLDRACK R A, NICHOLS T E, et al. Large-scale automated synthesis of human functional neuroimaging data[J]. Nature methods, 2011, 8(8): 665-670.
[107] BASSETT D S, SPORNS O. Network neuroscience[J]. Nature neuroscience, 2017, 20(3): 353-364.
[108] ROSENBAUM R, SMITH M A, KOHN A, et al. The spatial structure of correlated neuronal variability[J]. Nature neuroscience, 2017, 20(1): 107-114.
[109] VAN DEN HEUVEL M P, MANDL R C, KAHN R S, et al. Functionally linked resting-state networks reflect the underlying structural connectivity architecture of the human brain[J]. Human brain mapping, 2009, 30(10): 3127-3141.
[110] HERMUNDSTAD A M, BASSETT D S, BROWN K S, et al. Structural foundations of restingstate and task-based functional connectivity in the human brain[J]. Proceedings of the national academy of sciences, 2013, 110(15): 6169-6174.
[111] HAMMOND D K, VANDERGHEYNST P, GRIBONVAL R. Wavelets on graphs via spectral graph theory[J]. Applied and computational harmonic analysis, 2011, 30(2): 129-150.
[112] DEFFERRARD M, BRESSON X, VANDERGHEYNST P. Convolutional neural networks on graphs with fast localized spectral filtering[M]//Advances in neural information processing systems: volume 29. 2016: 3844-3852.
[113] ZHANG P, LAN C, XING J, et al. View adaptive neural networks for high performance skeletonbased human action recognition[J]. IEEE Transactions on pattern analysis and machine intelligence, 2019, 41(8): 1963-1978.
[114] ZHANG B, XIONG D, XIE J, et al. Neural machine translation with GRU-gated attention model [J]. IEEE Transactions on neural networks and learning systems, 2020, 31(11): 4688-4698.
[115] MELGANI F, BRUZZONE L. Classification of hyperspectral remote sensing images with support vector machines[J]. IEEE Transactions on geoscience and remote sensing, 2004, 42(8): 1778-1790.
[116] TOLSTIKHIN I O, HOULSBY N, KOLESNIKOV A, et al. Mlp-mixer: An all-mlp architecture for vision[C]//Advances in neural information processing systems: volume 34. 2021.
[117] TZOURIO-MAZOYER N, LANDEAU B, PAPATHANASSIOU D, et al. Automated anatomical labeling of activations in SPM using a macroscopic anatomical parcellation of the MNI MRI single-subject brain[J]. Neuroimage, 2002, 15(1): 273-289.
[118] GLASSER M F, COALSON T S, ROBINSON E C, et al. A multi-modal parcellation of human cerebral cortex[J]. Nature, 2016, 536(7615): 171-178.
[119] THOMAS YEO B, KRIENEN F M, SEPULCRE J, et al. The organization of the human cerebral cortex estimated by intrinsic functional connectivity[J]. Journal of neurophysiology, 2011, 106 (3): 1125-1165.
[120] KAMITANI Y, TONG F. Decoding the visual and subjective contents of the human brain[J]. Nature neuroscience, 2005, 8(5): 679-685.
[121] HAYNES J D, REES G. Decoding mental states from brain activity in humans[J]. Nature reviews neuroscience, 2006, 7(7): 523-534.
[122] SWISHER J D, GATENBY J C, GORE J C, et al. Multiscale pattern analysis of orientationselective activity in the primary visual cortex[J]. Journal of neuroscience, 2010, 30(1): 325-330.
[123] FANG M, TANG L, YANG X, et al. FTPG: A fine-grained traffic prediction method with graph attention network using big trace data[J]. IEEE Transactions on intelligent transportation systems, 2021.
[124] HUNTENBURG J M, BAZIN P L, MARGULIES D S. Large-scale gradients in human cortical organization[J]. Trends in cognitive sciences, 2018, 22(1): 21-31.
[125] FELLEMAN D J, VAN ESSEN D C. Distributed hierarchical processing in the primate cerebral cortex.[J]. Cerebral cortex, 1991, 1(1): 1-47.
[126] LI Y, LIANG R, WEI W, et al. Temporal pyramid network with spatial-temporal attention for pedestrian trajectory prediction[J]. IEEE Transactions on network science and engineering, 2021.
[127] WANG Y, ZHANG P, GAO S, et al. Pyramid spatial-temporal aggregation for video-based person re-identification[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2021: 12026-12035.
[128] WU B, LI J, YU J, et al. A survey of trustworthy graph learning: Reliability, explainability, and privacy protection[A]. 2022.
[129] LIU Q, FARAHIBOZORG S, PORCARO C, et al. Detecting large-scale networks in the human brain using high-density electroencephalography[J]. Human brain mapping, 2017, 38(9): 4631- 4643.
[130] PRETI M G, VAN DE VILLE D. Decoupling of brain function from structure reveals regional behavioral specialization in humans[J]. Nature communications, 2019, 10(1): 1-7.
[131] CURTIS C E, D’ESPOSITO M. Persistent activity in the prefrontal cortex during working memory[J]. Trends in cognitive sciences, 2003, 7(9): 415-423.
[132] ZHENG S, PUNIA D, WU H, et al. Graph theoretic analysis reveals intranasal oxytocin induced network changes over frontal regions[J]. Neuroscience, 2021, 459: 153-165.
[133] BAO P, SHE L, MCGILL M, et al. A map of object space in primate inferotemporal cortex[J]. Nature, 2020, 583(7814): 103-108.
[134] BALCONI M. Dorsolateral prefrontal cortex, working memory and episodic memory processes: insight through transcranial magnetic stimulation techniques[J]. Neuroscience bulletin, 2013, 29: 381-389.
[135] MARS R B, GROL M J. Dorsolateral prefrontal cortex, working memory, and prospective coding for action[J]. Journal of neuroscience, 2007, 27(8): 1801-1802.
[136] LIU D, GU X, ZHU J, et al. Medial prefrontal activity during delay period contributes to learning of a working memory task[J]. Science, 2014, 346(6208): 458-463.
[137] WIRT R A, HYMAN J M. Integrating spatial working memory and remote memory: interactions between the medial prefrontal cortex and hippocampus[J]. Brain sciences, 2017, 7(4): 43.
[138] TAKEDA K, FUNAHASHI S. Prefrontal task-related activity representing visual cue location or saccade direction in spatial working memory tasks[J]. Journal of neurophysiology, 2002, 87 (1): 567-588.
[139] HUNT P R, AGGLETON J P. Medial dorsal thalamic lesions and working memory in the rat [J]. Behavioral and neural biology, 1991, 55(2): 227-246.
[140] PARNAUDEAU S, O’NEILL P K, BOLKAN S S, et al. Inhibition of mediodorsal thalamus disrupts thalamofrontal connectivity and cognition[J]. Neuron, 2013, 77(6): 1151-1162.
[141] ACHESON A, RAY K L, HINES C S, et al. Functional activation and effective connectivity differences in adolescent marijuana users performing a simulated gambling task[J]. Journal of addiction, 2015, 2015.
[142] NOORI H R, LINAN A C, SPANAGEL R. Largely overlapping neuronal substrates of reactivity to drug, gambling, food and sexual cues: A comprehensive meta-analysis[J]. European Neuropsychopharmacology, 2016, 26(9): 1419-1430.
[143] IRIZAR P, ALBEIN-URIOS N, MARTÍNEZ-GONZÁLEZ J M, et al. Unpacking common and distinct neuroanatomical alterations in cocaine dependent versus pathological gambling[J]. European neuropsychopharmacology, 2020, 33: 81-88.
[144] ZHA R, LI P, LIU Y, et al. The orbitofrontal cortex represents advantageous choice in the Iowa gambling task[J]. Human brain mapping, 2022, 43(12): 3840-3856.
[145] HABIB R, DIXON M R. Neurobehavioral evidence for the “near-miss” effect in pathological gamblers[J]. Journal of the experimental analysis of behavior, 2010, 93(3): 313-328.
[146] BOUCHARD A E, DICKLER M, RENAULD E, et al. Concurrent transcranial direct current stimulation and resting-state functional magnetic resonance imaging in patients with gambling disorder[J]. Brain connectivity, 2021, 11(10): 815-821.
[147] SUN Y, YING H, SEETOHUL R M, et al. Brain fMRI study of crave induced by cue pictures in online game addicts (male adolescents)[J]. Behavioural brain research, 2012, 233(2): 563-576.
[148] CHIAO J Y, HARADA T, OBY E R, et al. Neural representations of social status hierarchy in human inferior parietal cortex[J]. Neuropsychologia, 2009, 47(2): 354-363.
[149] LAYCOCK R, CROSS A J, DALLE NOGARE F, et al. Self-rated social skills predict visual perception: impairments in object discrimination requiring transient attention associated with high autistic tendency[J]. Autism research, 2014, 7(1): 104-111.
[150] GREGORY M D, MERVIS C B, ELLIOTT M L, et al. Williams syndrome hemideletion and LIMK1 variation both affect dorsal stream functional connectivity[J]. Brain, 2019, 142(12): 3963-3974.
[151] ALLISON T, PUCE A, MCCARTHY G. Social perception from visual cues: role of the STS region[J]. Trends in cognitive sciences, 2000, 4(7): 267-278.
[152] COCCHI L, HALFORD G S, ZALESKY A, et al. Complexity in relational processing predicts changes in functional brain network dynamics[J]. Cerebral cortex, 2014, 24(9): 2283-2296.
[153] SUVILEHTO J T, RENVALL V, NUMMENMAA L. Relationship-specific encoding of social touch in somatosensory and insular cortices[J]. Neuroscience, 2021, 464: 105-116.
[154] DUMONTHEIL I, HOULTON R, CHRISTOFF K, et al. Development of relational reasoning during adolescence[J]. Developmental science, 2010, 13(6): F15-F24.
[155] WILSON B, MARSLEN-WILSON W D, PETKOV C I. Conserved sequence processing in primate frontal cortex[J]. Trends in neurosciences, 2017, 40(2): 72-82.
[156] HAUSER M F, HOFMANN J, OPITZ B. Rule and similarity in grammar: Their interplay and individual differences in the brain[J]. NeuroImage, 2012, 60(4): 2019-2026.
[157] ENGELEN T, DE GRAAF T A, SACK A T, et al. A causal role for inferior parietal lobule in emotion body perception[J]. Cortex, 2015, 73: 195-202.
[158] CAMACHO M C, KARIM H T, PERLMAN S B. Neural architecture supporting active emotion processing in children: A multivariate approach[J]. Neuroimage, 2019, 188: 171-180.
[159] VAN ZUTPHEN L, SIEP N, JACOB G A, et al. Impulse control under emotion processing: an fMRI investigation in borderline personality disorder compared to non-patients and cluster-C personality disorder patients[J]. Brain imaging and behavior, 2020, 14: 2107-2121.
[160] PALOMERO-GALLAGHER N, AMUNTS K. A short review on emotion processing: A lateralized network of neuronal networks[J]. Brain structure and function, 2022, 227(2): 673-684.
[161] POZZI E, SIMMONS J G, BOUSMAN C A, et al. The influence of maternal parenting style on the neural correlates of emotion processing in children[J]. Journal of the American academy of child and adolescent psychiatry, 2020, 59(2): 274-282.
[162] DUVAL E R, SHEYNIN J, KING A P, et al. Neural function during emotion processing and modulation associated with treatment response in a randomized clinical trial for posttraumatic stress disorder[J]. Depression and anxiety, 2020, 37(7): 670-681.
[163] GIL-DA COSTA R, BRAUN A, LOPES M, et al. Toward an evolutionary perspective on conceptual representation: species-specific calls activate visual and affective processing systems in the macaque[J]. Proceedings of the national academy of sciences, 2004, 101(50): 17516-17521.
[164] BUTLER P D, ABELES I Y, WEISKOPF N G, et al. Sensory contributions to impaired emotion processing in schizophrenia[J]. Schizophrenia bulletin, 2009, 35(6): 1095-1107.
[165] KNYAZEV G, SLOBODSKOJ-PLUSNIN J Y, BOCHAROV A. Event-related delta and theta synchronization during explicit and implicit emotion processing[J]. Neuroscience, 2009, 164 (4): 1588-1600.
[166] GOLDBERG H, PREMINGER S, MALACH R. The emotion–action link? Naturalistic emotional stimuli preferentially activate the human dorsal visual stream[J]. Neuroimage, 2014, 84: 254-264.
[167] TOZZI L, CARBALLEDO A, WETTERLING F, et al. Single-nucleotide polymorphism of the FKBP5 gene and childhood maltreatment as predictors of structural changes in brain areas involved in emotional processing in depression[J]. Neuropsychopharmacology, 2016, 41(2): 487-497.
[168] SCHNAKE T, EBERLE O, LEDERER J, et al. Higher-order explanations of graph neural networks via relevant walks[J]. IEEE Transactions on pattern analysis and machine intelligence, 2021(01): 1-1.
[169] BAJAJ M, CHU L, XUE Z Y, et al. Robust counterfactual explanations on graph neural networks [C]//Advances in neural information processing systems: volume 34. 2021: 5644-5655.
[170] LIN W, LAN H, WANG H, et al. Orphicx: A causality-inspired latent variable model for interpreting graph neural networks[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 13729-13738.
[171] LUCIC A, TER HOEVE M A, TOLOMEI G, et al. Cf-gnnexplainer: Counterfactual explanations for graph neural networks[C]//International conference on artificial intelligence and statistics. PMLR, 2022: 4499-4511.
[172] WU Y X, WANG X, ZHANG A, et al. Discovering invariant rationales for graph neural networks [C]//International conference on learning representations. 2022.
[173] WANG X, WU Y, ZHANG A, et al. Reinforced causal explainer for graph neural networks[J]. IEEE Transactions on pattern analysis and machine intelligence, 2022.
[174] SCHLICHTKRULL M S, DE CAO N, TITOV I. Interpreting graph neural networks for NLP with differentiable edge masking[C]//International conference on learning representations. 2020.
[175] YUAN J, RAN X, LIU K, et al. Machine Learning Applications on Neuroimaging for Diagnosis and Prognosis of Epilepsy: A Review[J]. Journal of neuroscience methods, 2021: 109441.
[176] HUTCHISON R M, WOMELSDORF T, ALLEN E A, et al. Dynamic functional connectivity: promise, issues, and interpretations[J]. Neuroimage, 2013, 80: 360-378.
[177] GONZALEZ-CASTILLO J, BANDETTINI P A. Task-based dynamic functional connectivity: Recent findings and open questions[J]. Neuroimage, 2018, 180: 526-533.
[178] ZHENG S, LIANG Z, QU Y, et al. Kuramoto model-based analysis reveals oxytocin effects on brain network dynamics[J]. International journal of neural systems, 2022, 32(02): 2250002.
[179] YING C, CAI T, LUO S, et al. Do transformers really perform badly for graph representation? [C]//Advances in neural information processing systems: volume 34. 2021: 28877-28888.
[180] SORGER B, DAHMEN B, REITHLER J, et al. Another kind of ‘BOLD Response’: answering multiple-choice questions via online decoded single-trial brain signals[J]. Progress in brain research, 2009, 177: 275-292.
[181] MIRKOVIC B, DEBENER S, JAEGER M, et al. Decoding the attended speech stream with multi-channel EEG: implications for online, daily-life applications[J]. Journal of neural engineering, 2015, 12(4): 046007.
[182] TAYEB Z, FEDJAEV J, GHABOOSI N, et al. Validating deep neural networks for online decoding of motor imagery movements from EEG signals[J]. Sensors, 2019, 19(1): 210.
[183] SU E, CAI S, XIE L, et al. STAnet: A spatiotemporal attention network for decoding auditory spatial attention from EEG[J]. IEEE Transactions on biomedical engineering, 2022, 69(7): 2233-2242.
[184] MARTINEZ-MONTES E, VALDÉS-SOSA P A, MIWAKEICHI F, et al. Concurrent EEG/fMRI analysis by multiway partial least squares[J]. NeuroImage, 2004, 22(3): 1023-1034.
[185] HUSTER R J, DEBENER S, EICHELE T, et al. Methods for simultaneous EEG-fMRI: an introductory review[J]. Journal of neuroscience, 2012, 32(18): 6053-6060.
[186] CICHY R M, OLIVA A. AM/EEG-fMRI fusion primer: resolving human brain responses in space and time[J]. Neuron, 2020, 107(5): 772-781.