基于运动想象的人机接口系统研究

人机交互是指人与外部设备，尤其是计算机之间的信息交互，键盘、鼠标、触控等都是常见的交互方式。然而，针对残疾人，如脑卒中（中风）、肌无力等患者， 这些交互方式可能就不再适用，因为他们无法执行出肢体的特定动作来完成对设备的控制。运动想象（MI）是一种用户想象自己肢体发生的动作而不做出实际执行的行为，其可以通过人体生物电信号监测出来。因此，基于运动想象行为过程中生物电信号的人机接口（HCI）为这些残疾人提供了另一种可行方案。本研究通过MI实验对用户脑电（EEG）和肌电（EMG）信号的变化及其相应的分类性能做出探索，并基于此建立了可以及时检测到用户MI活动的人机接口逻辑。

本研究设计了带有转变过程的抓握行为的动作执行（ME）和 MI 实验，招募了5名健康受试者完成该项实验并采集他们的肌电和脑电信号。针对目前的研究对于 MI 过程中肌电信号变化现象缺乏一致性观点的问题，本研究提取了三种状态（静息、ME、MI）下的10种肌电信号时域、频域特征，并对它们进行差异性分析后发现：MI 状态中的肌电出现轻微活动，较静息和ME状态都出现显著差异。针对如何在 MI 分类任务中有效利用两种生物电信号的问题，本研究对不同时间窗长度的肌电、脑电信号的不同特征在支持向量机（SVM）中表现出的分类性能做了比较，发现：短肌电信号即可完成预测任务，但这种分类性能在受试者间具备不稳定性；脑电信号通过共空间模式（CSP）提取的特征在分类器中的性能在受试者间表现较为稳定，但其信号长度需要达到2s才可以以较高性能完成预测任务。

为达成建立异步识别MI活动的人机接口逻辑的目的，本研究设计了开始时间随机化的单指抬起MI实验，招募了10名健康受试者完成了该项实验，并利用SVM分类器搭建了包含两步骤预测的人机接口系统逻辑。首先，利用肌电信号时域特征，在误差允许的范围内，以85%以上的准确率完成了受试者开始MI任务时间的预测。而在四种单指抬起MI任务的预测上，本研究提出了一种基于判别相关分析（DCA）的肌电时域特征和脑电CSP特征的融合分类方法，以接近70%的准确率完成四种MI任务的预测。最后，在总体的性能评估中，这种人机接口可以及时并准确地预测出受试者 57%试次中的 MI 任务。

Human-computer interface (HCI) refers to the exchange of information between humans and external devices, especially computers.Keyboard, mouse and touch screen are common ways to interact. However, for individuals with disabilities such as stroke and myasthenia gravis, these interaction methods may not be applicable, because they are unable to perform specific movements to control devices. Motor imagery (MI) is a behavior that users imagine themselves performing actions without actual execution, which can be detected by bioelectric signals.Therefore, HCI based on bioelectrical signals during MI behaviors provides another feasible option for these individuals.This study explored the changes in users' electroencephalography (EEG) and electromyography (EMG) signals during MI experiments and their corresponding classification performance. Based on this exploration, we established an HCI logic that can promptly detect users' MI activities.

This study designed motor execution (ME) and MI experiments with transitional grasping actions and recruited 5 healthy participants to complete the experiments while collecting their EMG and EEG signals. Addressing the inconsistency in current research regarding the changes in EMG signals during the MI process, this study extracted ten time and frequency domain features of EMG signals under three states (rest, ME, MI) and conducted a differential analysis. The results revealed that EMG exhibited slight activity during the MI state, showing significant differences compared to both rest and ME states.Regarding the effective utilization of both biosignals in MI prediction tasks, this study compared the classification performance of different features extracted from EMG and EEG signals of varying time window lengths using support vector machines (SVM). It was found that short-duration EMG signals were sufficient for prediction tasks, but this classification performance exhibited instability across participants. In contrast, EEG features extracted using common spatial patterns (CSP) demonstrated more stable performance across participants in the classifier, although their signal length needed to reach 2 seconds to achieve high-performance prediction tasks.

To achieve the goal of establishing an asynchronous recognition HCI logic for MI activities, this study designed an experiment involving single-finger lifting MI tasks with randomized start times.Ten healthy subjects were recruited to participate in the experiment.An HCI system logic comprising a two-step prediction process was constructed using an SVM classifier.Initially, using the EMG time-domain features, the prediction of the subjects' initiation time for MI tasks was achieved with an accuracy of over 85 percent within an acceptable error margin. Subsequently, for the prediction of the four types of single-finger lifting MI tasks, this study proposed a feature fusion classification method based on discriminant correlation analysis (DCA) of EMG time-domain features and EEG CSP features, which achieved a prediction accuracy close to 70 percent. Finally, in the overall performance evaluation,this HCI system can promptly and accurately predicted MI tasks in 57 percent of the trial instances for the participants.

[1] MYERS B A. A brief history of human-computer interaction technology[J]. interactions, 1998,5(2): 44-54.
[2] 何盛鸿. 基于 EEG 和 EOG 的异步人机接口及其应用[D]. 广州: 华南理工大学, 2017.
[3] ZHAO Y, ZHANG X, CHEN X, et al. Neuronal injuries in cerebral infarction and ischemic stroke: From mechanisms to treatment[J]. International journal of molecular medicine, 2022,49(2): 1-9.
[4] BINKS S, VINCENT A, PALACE J. Myasthenia gravis: a clinical-immunological update[J].Journal of neurology, 2016, 263: 826-834.
[5] ABIRI R, BORHANI S, SELLERS E W, et al. A comprehensive review of EEG-based brain–computer interface paradigms[J]. Journal of neural engineering, 2019, 16(1): 011001.
[6] BELKHIRIA C, BOUDIR A, HURTER C, et al. EOG-Based Human–Computer Interface:2000–2020 Review[J]. Sensors, 2022, 22(13): 4914.
[7] BERGEY G E, SQUIRES R D, SIPPLE W C. Electrocardiogram recording with pasteless electrodes[J]. IEEE transactions on biomedical engineering, 1971(3): 206-211.
[8] SARHAN S M, AL-FAIZ M Z, TAKHAKH A M. A review on EMG/EEG based control scheme of upper limb rehabilitation robots for stroke patients[J]. Heliyon, 2023.
[9] BELKHIRIA C, PEYSAKHOVICH V. Electro-encephalography and electro-oculography in aeronautics: A review over the last decade (2010–2020)[J]. Frontiers in Neuroergonomics,2020, 1: 606719.
[10] ACAR G, OZTURK O, GOLPARVAR A J, et al. Wearable and flexible textile electrodes for biopotential signal monitoring: A review[J]. Electronics, 2019, 8(5): 479.
[11] XIONG D, ZHANG D, ZHAO X, et al. Deep learning for EMG-based human-machine interaction: A review[J]. IEEE/CAA Journal of Automatica Sinica, 2021, 8(3): 512-533.
[12] BANIQUED P D E, STANYER E C, AWAIS M, et al. Brain–computer interface robotics for hand rehabilitation after stroke: A systematic review[J]. Journal of neuroengineering and rehabilitation, 2021, 18(1): 1-25.
[13] JEUNET C, GLIZE B, MCGONIGAL A, et al. Using EEG-based brain computer interface and neurofeedback targeting sensorimotor rhythms to improve motor skills: Theoretical background, applications and prospects[J]. Neurophysiologie Clinique, 2019, 49(2): 125-136.
[14] PADFIELD N, ZABALZA J, ZHAO H, et al. EEG-based brain-computer interfaces using motor-imagery: Techniques and challenges[J]. Sensors, 2019, 19(6): 1423.
[15] GASSER T, ROUSSON V, GASSER U S. EEG power and coherence in children with educational problems[J]. Journal of Clinical Neurophysiology, 2003, 20(4): 273-282.
[16] NIJHOLT A. BCI for games: A ‘state of the art’survey[C]//International Conference onEntertainment Computing. Springer, 2008: 225-228.
[17] WEN D, FAN Y, HSU S H, et al. Combining brain–computer interface and virtual reality for rehabilitation in neurological diseases: A narrative review[J]. Annals of physical and rehabilitation medicine, 2021, 64(1): 101404.
[18] MALEKI M, MANSHOURI N, KAYIKCIOGLU T. Brain-computer interface systems for smart homes-a review study[J]. Recent Advances in Electrical & Electronic Engineering (Formerly Recent Patents on Electrical & Electronic Engineering), 2021, 14(2): 144-155.
[19] WOLPAW J R, BIRBAUMER N, HEETDERKS W J, et al. Brain-computer interface technology: a review of the first international meeting[J]. IEEE transactions on rehabilitation engineering, 2000, 8(2): 164-173.
[20] NICOLAS-ALONSO L F, GOMEZ-GIL J. Brain computer interfaces, a review[J]. sensors,2012, 12(2): 1211-1279.
[21] KAPGATE D, KALBANDE D. A review on visual brain computer interface[C]//Advancements of Medical Electronics: Proceedings of the First International Conference, ICAME 2015.Springer, 2015: 193-206.
[22] NIJBOER F, FURDEA A, GUNST I, et al. An auditory brain–computer interface (BCI)[J].Journal of neuroscience methods, 2008, 167(1): 43-50.
[23] YAO L, SHENG X, ZHANG D, et al. A BCI system based on somatosensory attentional orientation[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2016, 25(1):81-90.
[24] FABIANI M, GRATTON G, KARIS D, et al. Definition, identification, and reliability of measurement of the P300 component of the event-related brain potential[J]. Advances in psychophysiology, 1987, 2(S1): 78.
[25] FAZEL-REZAI R, AMIRI S, RABBI A, et al. A Review of P300, SSVEP, and Hybrid P300/SSVEP Brain-Computer Interface Systems[Z]. 2013.
[26] 王攀, 沈继忠, 施锦河. 基于小波变换和时域能量熵的 P300 特征提取算法[J]. 仪器仪表学报, 2011, 32(6): 1284-1289.
[27] TAKANO K, KOMATSU T, HATA N, et al. Visual stimuli for the P300 brain–computer interface: a comparison of white/gray and green/blue flicker matrices[J]. Clinical neurophysiology,2009, 120(8): 1562-1566.
[28] WANG Y, CHEN X, GAO X, et al. A benchmark dataset for SSVEP-based brain–computer interfaces[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2016, 25(10): 1746-1752.
[29] HERRMANN C S. Human EEG responses to 1–100 Hz flicker: resonance phenomena in visual cortex and their potential correlation to cognitive phenomena[J]. Experimental brain research, 2001, 137: 346-353.
[30] KUNDU S, ARI S. P300 based character recognition using convolutional neural network and support vector machine[J]. Biomedical Signal Processing and Control, 2020, 55: 101645.
[31] ZHANG Y, ZHOU G, JIN J, et al. Frequency recognition in SSVEP-based BCI using multiset canonical correlation analysis[J]. International journal of neural systems, 2014, 24(04):1450013.
[32] NAKANISHI M, WANG Y, WANG Y T, et al. A high-speed brain speller using steady-state visual evoked potentials[J]. International journal of neural systems, 2014, 24(06): 1450019.
[33] RASHID M, SULAIMAN N, PP ABDUL MAJEED A, et al. Current status, challenges, and possible solutions of EEG-based brain-computer interface: a comprehensive review[J]. Frontiers in neurorobotics, 2020, 14: 515104.
[34] MORASH V, BAI O, FURLANI S, et al. Classifying EEG signals preceding right hand, left hand, tongue, and right foot movements and motor imageries[J]. Clinical neurophysiology,2008, 119(11): 2570-2578.
[35] MCFARLAND D J, WOLPAW J R. Sensorimotor rhythm-based brain-computer interface (BCI): feature selection by regression improves performance[J]. IEEE transactions on neural systems and rehabilitation engineering, 2005, 13(3): 372-379.
[36] 周思捷, 白红民. 事件相关去同步化和同步化方法在脑电信号分析中的研究进展[J]. 中国微侵袭神经外科杂志, 2018, 23(3): 141-143.
[37] GAO H, LUO L, PI M, et al. EEG-based volitional control of prosthetic legs for walking in different terrains[J]. IEEE Transactions on Automation Science and Engineering, 2019, 18(2):530-540.
[38] LAFLEUR K, CASSADY K, DOUD A, et al. Quadcopter control in three-dimensional space using a noninvasive motor imagery-based brain–computer interface[J]. Journal of neural engineering, 2013, 10(4): 046003.
[39] 岑尧辉. 在线异步 MI-BCI 系统实验范式设计[D]. 哈尔滨工业大学, 2019.
[40] GANDEVIA S C, WILSON L R, INGLIS J T, et al. Mental rehearsal of motor tasks recruits alpha-motoneurones but fails to recruit human fusimotor neurones selectively.[J]. The Journal of physiology, 1997, 505(Pt 1): 259.
[41] GUILLOT A, LEBON F, ROUFFET D, et al. Muscular responses during motor imagery as a function of muscle contraction types[J]. International Journal of Psychophysiology, 2007, 66(1): 18-27.
[42] PERSONNIER P, BALLAY Y, PAPAXANTHIS C. Mentally represented motor actions in normal aging: III. Electromyographic features of imagined arm movements[J]. Behavioural brain research, 2010, 206(2): 184-191.
[43] LEBON F, ROUFFET D, COLLET C, et al. Modulation of EMG power spectrum frequency during motor imagery[J]. Neuroscience letters, 2008, 435(3): 181-185.
[44] GENTILI R, PAPAXANTHIS C, POZZO T. Improvement and generalization of arm motor performance through motor imagery practice[J]. Neuroscience, 2006, 137(3): 761-772.
[45] WIERZGAŁA P, ZAPAŁA D, WOJCIK G M, et al. Most popular signal processing methods in motor-imagery BCI: a review and meta-analysis[J]. Frontiers in neuroinformatics, 2018, 12:78.
[46] HADJIAROS M, NEOKLEOUS K, SHIMI A, et al. Virtual Reality Cognitive Gaming based on Brain Computer Interfacing: A narrative review[J]. IEEE Access, 2023.
[47] 潘家辉. 基于 P300 和 SSVEP 的高性能脑机接口及其应用研究[D]. 华南理工大学, 2014.
[48] BRUNNER C, LEEB R, MÜLLER-PUTZ G, et al. BCI Competition 2008–Graz data set A[J].Institute for Knowledge Discovery (Laboratory of Brain-Computer Interfaces), Graz University of Technology, 2008, 16: 1-6.
[49] LEEB R, BRUNNER C, MÜLLER-PUTZ G, et al. BCI Competition 2008–Graz data set B[J]. Graz University of Technology, Austria, 2008, 16: 1-6.
[50] SCHALK G, MCFARLAND D J, HINTERBERGER T, et al. BCI2000: a general-purpose brain-computer interface (BCI) system[J]. IEEE Transactions on biomedical engineering, 2004,51(6): 1034-1043.
[51] ANG K K, CHIN Z Y, WANG C, et al. Filter bank common spatial pattern algorithm on BCI competition IV datasets 2a and 2b[J]. Frontiers in neuroscience, 2012, 6: 39.
[52] JIN Z, ZHOU G, GAO D, et al. EEG classification using sparse Bayesian extreme learning machine for brain–computer interface[J]. Neural Computing and Applications, 2020, 32: 6601-6609.
[53] HA K W, JEONG J W. Motor imagery EEG classification using capsule networks[J]. Sensors,2019, 19(13): 2854.
[54] ZHANG Y, NAM C S, ZHOU G, et al. Temporally constrained sparse group spatial patterns for motor imagery BCI[J]. IEEE transactions on cybernetics, 2018, 49(9): 3322-3332.
[55] CHO H, AHN M, AHN S, et al. Supporting data for “EEG datasets for motor imagery brain computer interface.”[J]. GigaScience Database, 2017.
[56] BI L, GUAN C, et al. A review on EMG-based motor intention prediction of continuous human upper limb motion for human-robot collaboration[J]. Biomedical Signal Processing and Control, 2019, 51: 113-127.
[57] BOTROS F S, PHINYOMARK A, SCHEME E J. Electromyography-based gesture recognition: Is it time to change focus from the forearm to the wrist?[J]. IEEE Transactions on Industrial Informatics, 2020, 18(1): 174-184.
[58] MOMEN K, KRISHNAN S, CHAU T. Real-time classification of forearm electromyographic signals corresponding to user-selected intentional movements for multifunction prosthesis control[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2007, 15(4):535-542.
[59] 许家超. 基于运动想象脑电信号的游戏控制研究[D]. 哈尔滨工业大学, 2020.
[60] PEIRCE J W. PsychoPy—psychophysics software in Python[J]. Journal of neuroscience methods, 2007, 162(1-2): 8-13.
[61] TANG C, XU L, CHEN P, et al. A Novel Multiple Motor Imagery Experimental Paradigm Design and Neural Decoding[C]//2020 Chinese Automation Congress (CAC). IEEE, 2020:4024-4028.
[62] 李翔. 基于运动想象的在线脑-机接口系统的研究与实现[D]. 清华大学, 2012.
[63] 张树丰. 负载和运动速度对肌疲劳 sEMG 特征分类的影响分析[D]. 天津大学, 2020.
[64] 毛琳. 肌电, 脑电信号在上肢康复中的研究与应用[D]. 中国电子科技集团公司电子科学研究院, 2022.
[65] DE LUCA C J, GILMORE L D, KUZNETSOV M, et al. Filtering the surface EMG signal:Movement artifact and baseline noise contamination[J]. Journal of biomechanics, 2010, 43(8):1573-1579.
[66] HARTMANN M, SCHINDLER K, GEBBINK T A, et al. PureEEG: automatic EEG artifact removal for epilepsy monitoring[J]. Neurophysiologie Clinique/Clinical Neurophysiology, 2014,44(5): 479-490.
[67] JIANG X, BIAN G B, TIAN Z. Removal of artifacts from EEG signals: a review[J]. Sensors,2019, 19(5): 987.
[68] 骆睿鹏, 冯铭科, 黄鑫, 等. 脑电信号预处理方法研究综述.[J]. Electronic Science & Technology, 2023, 36(4).
[69] URIGÜEN J A, GARCIA-ZAPIRAIN B. EEG artifact removal—state-of-the-art and guidelines[J]. Journal of neural engineering, 2015, 12(3): 031001.
[70] JUNG T P, MAKEIG S, HUMPHRIES C, et al. Removing electroencephalographic artifacts by blind source separation[J]. Psychophysiology, 2000, 37(2): 163-178.
[71] WINTER W R, NUNEZ P L, DING J, et al. Comparison of the effect of volume conduction on EEG coherence with the effect of field spread on MEG coherence[J]. Statistics in medicine, 2007, 26(21): 3946-3957.
[72] PION-TONACHINI L, KREUTZ-DELGADO K, MAKEIG S. ICLabel: An automated electroencephalographic independent component classifier, dataset, and website[J]. NeuroImage,2019, 198: 181-197.
[73] PHINYOMARK A, QUAINE F, CHARBONNIER S, et al. EMG feature evaluation for improving myoelectric pattern recognition robustness[J]. Expert Systems with applications, 2013,40(12): 4832-4840.
[74] ASGHAR A, JAWAID KHAN S, AZIM F, et al. Review on electromyography based intention for upper limb control using pattern recognition for human-machine interaction[J]. Proceedings of the Institution of Mechanical Engineers, Part H: Journal of Engineering in Medicine, 2022,236(5): 628-645.
[75] JARAMILLO A G, BENALCAZAR M E. Real-time hand gesture recognition with EMG using machine learning[C]//2017 IEEE second ecuador technical chapters meeting (ETCM). IEEE,2017: 1-5.
[76] CHANG J, PHINYOMARK A, BATEMAN S, et al. Wearable emg-based gesture recognition systems during activities of daily living: An exploratory study[C]//2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). IEEE,2020: 3448-3451.
[77] PHINYOMARK A, PHUKPATTARANONT P, LIMSAKUL C. Feature reduction and selection for EMG signal classification[J]. Expert systems with applications, 2012, 39(8): 7420-7431.
[78] KIM K S, CHOI H H, MOON C S, et al. Comparison of k-nearest neighbor, quadratic discriminant and linear discriminant analysis in classification of electromyogram signals based on the wrist-motion directions[J]. Current applied physics, 2011, 11(3): 740-745.
[79] JOLLIFFE I T, CADIMA J. Principal component analysis: a review and recent developments[J]. Philosophical transactions of the royal society A: Mathematical, Physical and Engineering Sciences, 2016, 374(2065): 20150202.
[80] CORTES C, VAPNIK V. Support-vector networks[J]. Machine learning, 1995, 20: 273-297.
[81] DEKA P C, et al. Support vector machine applications in the field of hydrology: a review[J].Applied soft computing, 2014, 19: 372-386.
[82] MARIS E, OOSTENVELD R. Nonparametric statistical testing of EEG-and MEG-data[J].Journal of neuroscience methods, 2007, 164(1): 177-190.
[83] SASSENHAGEN J, DRASCHKOW D. Cluster-based permutation tests of MEG/EEG data do not establish significance of effect latency or location[J]. Psychophysiology, 2019, 56(6):e13335.
[84] KOLES Z J, LAZAR M S, ZHOU S Z. Spatial patterns underlying population differences in the background EEG[J]. Brain topography, 1990, 2: 275-284.
[85] WANG F, LIU H, ZHAO L, et al. Improved brain–computer interface signal recognition algorithm based on few-channel motor imagery[J]. Frontiers in Human Neuroscience, 2022, 16:880304.
[86] ROSS A, JAIN A. Information fusion in biometrics[J]. Pattern recognition letters, 2003, 24(13): 2115-2125.
[87] ROSS A, JAIN A K. Multimodal biometrics: An overview[C]//2004 12th European signal processing conference. IEEE, 2004: 1221-1224.
[88] HAGHIGHAT M, ABDEL-MOTTALEB M, ALHALABI W. Discriminant correlation analysis for feature level fusion with application to multimodal biometrics[C]//2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2016: 1866-1870.
[89] HAGHIGHAT M, ABDEL-MOTTALEB M, ALHALABI W. Discriminant correlation analysis: Real-time feature level fusion for multimodal biometric recognition[J]. IEEE Transactions on Information Forensics and Security, 2016, 11(9): 1984-1996.
[90] AL-QURAISHI M S, ELAMVAZUTHI I, TANG T B, et al. Multimodal fusion approach based on EEG and EMG signals for lower limb movement recognition[J]. IEEE Sensors Journal,2021, 21(24): 27640-27650.
[91] 廖培伶, 周凤坤, 吴李硕, 等. 计算机辅助认知康复系统对卒中后认知障碍患者认知功能的影响[J]. 微创医学, 2022, 17(05): 645-648.
[92] YUAN H, LI Y, YANG J, et al. State of the Art of Non-Invasive Electrode Materials for Brain–Computer Interface[J]. Micromachines, 2021, 12(12): 1521.