Investigation of neural network approaches for unified spectral and prosodic feature enhancement

Lin，Wei Cheng1; Tsao，Yu1; Chen，Fei2; Wang，Hsin Min3

10.1109/APSIPAASC47483.2019.9023309

2019-11-01

2309-9402

978-1-7281-3249-5

2019 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2019

1179-1184

18-21 Nov. 2019

Lanzhou, China

345 E 47TH ST, NEW YORK, NY 10017 USA

IEEE

Most speech enhancement (SE) systems focus on the spectral feature or raw-waveform enhancement. However, many speech-related applications rely on other features rather than the spectral features, such as the intensity and fundamental frequency (f0). Therefore, a unified feature enhancement for different types of features is worth investigating. In this work, we train our neural network (NN)-based SE system in a manner that simultaneously minimizes the spectral loss and preserves the correctness of the intensity and f0 contours extracted from the enhanced speech. The idea is to introduce an NN-based feature extractor to the SE framework that imitates the feature extraction of Praat. Then, we can train the SE system by minimizing the combined loss of the spectral feature, intensity, and f0. We investigate three bidirectional long short-term memory (BLSTM)-based unified feature enhancement systems: fixed-concat, joint-concat, and multi-task. The results of the experiments on the Taiwan Mandarin hearing in a noise test dataset (TMHINT) demonstrate that all three systems show improved intensity and f0 extraction accuracy without sacrificing the perceptual evaluation of the speech quality and short-time objective intelligibility scores compared with the baseline SE system. Further analysis of the experimental results shows that the improvement mostly comes from better f0 contours under difficult conditions such as low signal-to-noise ratio and nonstationary noises. Our work demonstrates the advantage of the unified feature enhancement and provides new insights for SE.

Feature extraction Iron Speech enhancement Artificial neural networks Noise measurement Training Mathematical model

其他

英语

CPCI-S ; EI

MOST Taiwan Grants[108-2634-F-008-004][108-2634-F-001-004]

Computer Science ; Engineering ; Imaging Science & Photographic Technology

Computer Science, Software Engineering ; Engineering, Electrical & Electronic ; Imaging Science & Photographic Technology

WOS:000555696900199

20201308362137

Extraction ; Quality control ; Speech intelligibility ; Signal to noise ratio ; Audition ; Statistical tests

Ergonomics and Human Factors Engineering:461.4 ; Information Theory and Signal Processing:716.1 ; Speech:751.5 ; Chemical Operations:802.3 ; Quality Assurance and Control:913.3 ; Mathematical Statistics:922.2 ; Acoustical Instruments:941.1

2-s2.0-85082398079

Scopus

1.Research Center for Information Technology Innovation,Academic Sinica,Taiwan
2.Southern University of Science and Technology,Department of Electrical and Electronic Engineering,China
3.Institute of Information Science,Academic Sinica,Taiwan

Lin，Wei Cheng,Tsao，Yu,Chen，Fei,et al. Investigation of neural network approaches for unified spectral and prosodic feature enhancement[C]. 345 E 47TH ST, NEW YORK, NY 10017 USA:IEEE,2019:1179-1184.