A Multi-Stage Adaptive Feature Fusion Neural Network for Multimodal Gait Recognition

Shinan Zou1; Jianbo Xiong1; Chao Fan2; Chuanfu Shen2; Shiqi Yu2; Jin Tang1

10.1109/TBIOM.2024.3384704

2024

IEEE International Joint Conference on Biometrics (IJCB)

2637-6407

IEEE Transactions on Biometrics, Behavior, and Identity Science

PP

99

1-1

SEP 25-28, 2023

null,Ljubljana,SLOVENIA

345 E 47TH ST, NEW YORK, NY 10017 USA

IEEE

Gait recognition is a biometric technology that has received extensive attention. Most existing gait recognition algorithms are unimodal, and a few multimodal gait recognition algorithms perform multimodal fusion only once. None of these algorithms may fully exploit the complementary advantages of the multiple modalities. In this paper, by considering the temporal and spatial characteristics of gait data, we propose a multi-stage feature fusion strategy (MSFFS), which performs multimodal fusions at different stages in the feature extraction process. Also, we propose an adaptive feature fusion module (AFFM) that considers the semantic association between silhouettes and skeletons. The fusion process fuses different silhouette areas with their more related skeleton joints. Since visual appearance changes and time passage co-occur in a gait period, we propose a multiscale spatial-temporal feature extractor (MSSTFE) to learn the spatial-temporal linkage features thoroughly. Specifically, MSSTFE extracts and aggregates spatial-temporal linkages information at different spatial scales. Combining the strategy and modules mentioned above, we propose a multi-stage adaptive feature fusion (MSAFF) neural network, which shows state-of-the-art performance in many experiments on three datasets. Besides, MSAFF is equipped with feature dimensional pooling (FD Pooling), which can significantly reduce the dimension of the gait representations without hindering the accuracy.

Gait recognition multimodal silhouette skeleton

其他

英语

CPCI-S ; EI

Computer Science ; Imaging Science & Photographic Technology

Computer Science, Artificial Intelligence ; Imaging Science & Photographic Technology

WOS:001180818700103

IEEE

1.School of Automation, Central South University, China
2.Department of Computer Science and Engineering, Southern University of Science and Technology, China

Shinan Zou,Jianbo Xiong,Chao Fan,et al. A Multi-Stage Adaptive Feature Fusion Neural Network for Multimodal Gait Recognition[C]. 345 E 47TH ST, NEW YORK, NY 10017 USA:IEEE,2024:1-1.