Multimodal Sensing for Depression Risk Detection: Integrating Audio, Video, and Text Data

Zhang, Zhenwei1,2; Zhang, Shengming3; Ni, Dong1,2; Wei, Zhaoguo4,5; Yang, Kongjun4,5; Jin, Shan4,5; Huang, Gan1,2; Liang, Zhen1,2; Zhang, Li1,2; Li, Linling1,2; Ding, Huijun1,2; Zhang, Zhiguo6,7; Wang, Jianhong4,5

2024-06-01

10.3390/s24123714

SENSORS   影响因子和分区

1424-8220

Depression is a major psychological disorder with a growing impact worldwide. Traditional methods for detecting the risk of depression, predominantly reliant on psychiatric evaluations and self-assessment questionnaires, are often criticized for their inefficiency and lack of objectivity. Advancements in deep learning have paved the way for innovations in depression risk detection methods that fuse multimodal data. This paper introduces a novel framework, the Audio, Video, and Text Fusion-Three Branch Network (AVTF-TBN), designed to amalgamate auditory, visual, and textual cues for a comprehensive analysis of depression risk. Our approach encompasses three dedicated branches-Audio Branch, Video Branch, and Text Branch-each responsible for extracting salient features from the corresponding modality. These features are subsequently fused through a multimodal fusion (MMF) module, yielding a robust feature vector that feeds into a predictive modeling layer. To further our research, we devised an emotion elicitation paradigm based on two distinct tasks-reading and interviewing-implemented to gather a rich, sensor-based depression risk detection dataset. The sensory equipment, such as cameras, captures subtle facial expressions and vocal characteristics essential for our analysis. The research thoroughly investigates the data generated by varying emotional stimuli and evaluates the contribution of different tasks to emotion evocation. During the experiment, the AVTF-TBN model has the best performance when the data from the two tasks are simultaneously used for detection, where the F1 Score is 0.78, Precision is 0.76, and Recall is 0.81. Our experimental results confirm the validity of the paradigm and demonstrate the efficacy of the AVTF-TBN model in detecting depression risk, showcasing the crucial role of sensor-based data in mental health detection.

depression risk detection emotion elicitation paradigm multimodal data

SCI ; EI

英语

其他

Shenzhen Science and Technology Research and Development Fund for Sustainable Development Project[KCXFZ20201221173613036] ; Medical Scientific Research Foundation of Guangdong Province of China[B2023078] ; Shenzhen Soft Science Research Program Project[RKX20220705152815035]

Chemistry ; Engineering ; Instruments & Instrumentation

Chemistry, Analytical ; Engineering, Electrical & Electronic ; Instruments & Instrumentation

WOS:001255873400001

MDPI

CHEMISTRY

Web of Science

1.Shenzhen Univ, Hlth Sci Ctr, Sch Biomed Engn, Shenzhen 518060, Peoples R China
2.Natl Reg Key Technol Engn Lab Med Ultrasound, Guangdong Prov Key Lab Biomed Measurements & Ultra, Shenzhen 518060, Peoples R China
3.Southern Univ Sci & Technol, Affiliated Mental Hlth Ctr, Shenzhen 518055, Peoples R China
4.Shenzhen Kangning Hosp, Dept Clin Psychol, Shenzhen 518020, Peoples R China
5.Shenzhen Mental Hlth Ctr, Shenzhen 518020, Peoples R China
6.Harbin Inst Technol, Sch Comp Sci & Technol, Shenzhen 518055, Peoples R China
7.Peng Cheng Lab, Shenzhen 518055, Peoples R China

Zhang, Zhenwei,Zhang, Shengming,Ni, Dong,et al. Multimodal Sensing for Depression Risk Detection: Integrating Audio, Video, and Text Data[J]. SENSORS,2024,24(12).

Zhang, Zhenwei.,Zhang, Shengming.,Ni, Dong.,Wei, Zhaoguo.,Yang, Kongjun.,...&Wang, Jianhong.(2024).Multimodal Sensing for Depression Risk Detection: Integrating Audio, Video, and Text Data.SENSORS,24(12).

Zhang, Zhenwei,et al."Multimodal Sensing for Depression Risk Detection: Integrating Audio, Video, and Text Data".SENSORS 24.12(2024).