南方科技大学知识苑(SUSTech KC): WISE: Word-level interaction-based multimodal fusion for speech emotion recognition

题名	WISE: Word-level interaction-based multimodal fusion for speech emotion recognition
作者	Shen，Guang 1; Lai，Riwei 1; Chen，Rui 1; Zhang，Yu2 ; Zhang，Kejia 1; Han，Qilong 1; Song，Hongtao 1
通讯作者	Chen，Rui
DOI	10.21437/Interspeech.2020-3131
发表日期	2020
ISSN	2308-457X
EISSN	1990-9772
会议录名称	Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH
卷号	2020-October
页码	369-373
摘要	While having numerous real-world applications, speech emotion recognition is still a technically challenging problem. How to effectively leverage the inherent multiple modalities in speech data (e.g., audio and text) is key to accurate classification. Existing studies normally choose to fuse multimodal features at the utterance level and largely neglect the dynamic interplay of features from different modalities at a fine-granular level over time. In this paper, we explicitly model dynamic interactions between audio and text at the word level via interaction units between two long short-term memory networks representing audio and text. We also devise a hierarchical representation of audio information from the frame, phoneme and word levels, which largely improves the expressiveness of resulting audio features. We finally propose WISE, a novel wordlevel interaction-based multimodal fusion framework for speech emotion recognition, to accommodate the aforementioned components. We evaluate WISE on the public benchmark IEMOCAP corpus and demonstrate that it outperforms state-of-the-art methods.
关键词	Deep multimodal fusion Dynamic interaction mechanism Hierarchical representation Speech emotion recognition
学校署名	其他
语种	英语
相关链接	[Scopus记录]
收录类别	EI
EI入藏号	20205209692343
EI主题词	Speech communication ; Emotion Recognition
EI分类号	Data Processing and Image Processing:723.2 ; Speech:751.5
Scopus记录号	2-s2.0-85098107351
来源库	Scopus
引用统计	被引频次[WOS]：11
成果类型	会议论文
条目标识符	http://sustech.caswiz.com/handle/2SGJ60CL/210964
专题	工学院_计算机科学与工程系
作者单位	1.College of Computer Science and Technology,Harbin Engineering University,Harbin,China 2.Department of Computer Science and Engineering,Southern University of Science and Technology,Shenzhen,China
推荐引用方式 GB/T 7714	Shen，Guang,Lai，Riwei,Chen，Rui,et al. WISE: Word-level interaction-based multimodal fusion for speech emotion recognition[C],2020:369-373.