Cross-Subject Classification of Spoken Mandarin Vowels and Tones with EEG Signals: A Study of End-to-End CNN with Fine-Tuning

Xinyu Wang1; Mingtao Li1; Hao Li1; Sio Hang Pun2; Fei Chen1

10.1109/APSIPAASC58517.2023.10317100

2023-10-31

Asia-Pacific-Signal-and-Information-Processing-Association Annual Summit and Conference (APSIPA ASC)

2640-009X

979-8-3503-0068-0

2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)

535-539

31 Oct.-3 Nov. 2023

Taipei, Taiwan

345 E 47TH ST, NEW YORK, NY 10017 USA

IEEE

Direct speech brain-computer interface (DS-BCI) is an ideal way for speech communication by decoding signals collected from the brain. Electroencephalogram (EEG) has gained widespread use in DS-BCI studies due to its simplicity of operation and high temporal resolution. However, as human brain exhibits considerable inter-individual variability, classification models trained on the basis of data from one subject may not generalise well to other individuals, which is a major challenge in existing EEG signal classification studies. In this paper, the cross-subject classification performance of spoken Mandarin speech with EEG signals was investigated by using an end-to-end convolutional neural network (CNN) model pre-trained on the source data and fine-tuned on the target data. The raw EEG signals were directly used as the input to the model without using extracted features. In addition, adding Gaussian noise was used as the data augmentation method in order to deal with the unbalanced dataset. The proposed method was tested on a collected EEG dataset of spoken Mandarin speech, including vowel classification and tone classification tasks. The average classification accuracies of four vowels and four tones were 63.1% and 51.7% respectively. The average accuracy of tone classification was significantly improved compared with the machine learning and subject-dependent methods. The results of this work showed the potential of the fine-tuning based CNN model in the cross-subject studies of EEG decoding.

Pattern classification Oral communication Brain modeling Feature extraction Electroencephalography Data models Decoding

第一

英语

EI ; CPCI-S

National Natural Science Foundation of China[61971212]

Computer Science

Computer Science, Artificial Intelligence ; Computer Science, Information Systems ; Computer Science, Theory & Methods

WOS:001108741800085

20235115257048

Biomedical signal processing ; Brain computer interface ; Convolutional neural networks ; Decoding ; Electroencephalography ; Gaussian noise (electronic) ; Linguistics ; Neural network models ; Speech communication ; Tuning

Medicine and Pharmacology:461.6 ; Information Theory and Signal Processing:716.1 ; Computer Peripheral Equipment:722.2 ; Data Processing and Image Processing:723.2 ; Artificial Intelligence:723.4 ; Speech:751.5 ; Information Sources and Analysis:903.1

IEEE

1.Southern University of Science and Technology, Shenzhen, China
2.University of Macau, Macau, China

Xinyu Wang,Mingtao Li,Hao Li,et al. Cross-Subject Classification of Spoken Mandarin Vowels and Tones with EEG Signals: A Study of End-to-End CNN with Fine-Tuning[C]. 345 E 47TH ST, NEW YORK, NY 10017 USA:IEEE,2023:535-539.