SML: A Skeleton-based multi-feature learning method for sign language recognition

Deng，Zhiwen1,2; Leng，Yuquan3; Hu，Jing1; Lin，Zengrong1; Li，Xuerui4; Gao，Qing1

2024-10-09

10.1016/j.knosys.2024.112288

Knowledge-Based Systems   影响因子和分区

0950-7051

Sign language recognition (SLR) is an effective solution to communication barriers experienced by hearing and vocally impaired individuals with other communities. Its applications extend to human–robot interaction (HRI), virtual reality (VR), and augmented reality (AR). However, the diverse nature of sign languages, stemming from varying user habits and geographical regions, poses significant challenges. To address these challenges, we propose the Skeleton-based Multi-feature Learning method (SML). This method comprises a Multi Feature Aggregation (MFA) module, designed to capture the inherent relationships between different skeleton-based features, enabling effective fusion of complementary information. Furthermore, we propose the Self knowledge distillation Guided Adaptive Residual Decoupled Graph Convolutional Network (SGAR-DGCN) for feature extraction. SGAR-DGCN consists of three components: a Self Knowledge Distillation (SKD) mechanism to enhance model training, convergence, and accuracy; a DGCN-Block, incorporating Decoupled GCN and Spatio Temporal Channel attention (STC) for efficient feature extraction; and an Adaptive Residual Block (ARes-Block) for cross-layer information fusion. Experimental results demonstrate that our SML method outperforms state-of-the-art approaches on the WLASL (55.85%) and AUTSL (96.85%) datasets, solely utilizing skeleton data. Code is available at https://github.com/DzwFine37/SML.

Adaptive residual Multi-feature aggregation Sign language recognition Skeleton

SCI ; EI

英语

其他

COMPUTER SCIENCE

2-s2.0-85200267721

Scopus

1.School of Electronics and Communication Engineering,Shenzhen Campus of Sun Yat-sen University,Shenzhen,518107,China
2.School of Communication and Information Engineering,Chongqing University of Posts and Telecommunications,Chongqing,400065,China
3.Department of Mechanical and Energy Engineering,Southern University of Science and Technology,Shenzhen,518055,China
4.Department of Computer Science and Engineering,University at Buffalo,Buffalo,14221,United States

Deng，Zhiwen,Leng，Yuquan,Hu，Jing,et al. SML: A Skeleton-based multi-feature learning method for sign language recognition[J]. Knowledge-Based Systems,2024,301.

Deng，Zhiwen,Leng，Yuquan,Hu，Jing,Lin，Zengrong,Li，Xuerui,&Gao，Qing.(2024).SML: A Skeleton-based multi-feature learning method for sign language recognition.Knowledge-Based Systems,301.

Deng，Zhiwen,et al."SML: A Skeleton-based multi-feature learning method for sign language recognition".Knowledge-Based Systems 301(2024).