Dual branch feature fusion network based gait recognition algorithm 双分支特征融合网络的步态识别算法

Xu，Shuo1; Zheng，Feng2; Tang，Jun1; Bao，Wenxia1

2022-07-16

10.11834/jig.200730

中国图象图形学报   影响因子和分区

1006-8961

Objective Gait is a kind of human walking pattern, which is one of the key biometric features for person identification. As a non-contact and long-distance recognition way to capture human identity information, gait recognition has been developed in video surveillance and public security. Gait recognition algorithms can be segmented into two mainstreams like appearance-based methods and the model-based methods. The appearance-based methods extract gait from a sequence of silhouette images in common. However, the appearance-based methods are basically affected by appearance changes like non-rigid clothing deformation and background clutters. Different from the appearance-based methods, the model-based methods commonly leverage body structure or motion prior to model gait pattern and more robust to appearance variations. Actually, it is challenged to identify a universal model for gait description, and the previous pre-defined models can be constrained in certain scenarios. Recent model-based methods are focused on deep learning-based pose estimation to model key-points of human body. But the estimated pose model constrains the redundant noises in subject to pose estimators and occlusion. In summary, the appearance-based methods are based visual features description while the model-based methods tend to describe a semantic level-based motion and structure. We aim to design a novel approach for gait recognition beyond the existed two methods mentioned above and improve gait recognition ability via the added appearance features and pose features. Method we design a dual-branch network for gait recognition. The input data are fed into a dual-branch network to extract appearance features and pose features each. Then, the two kinds of features are merged into the final gait features in the context of feature fusion module. In detail, we adopt an optimal network GaitSet as the appearance branch to extract appearance features from silhouette images and design a two-stream convolutional neural network (CNN) to extract pose features from pose key-points based on the position information and motion information. Meanwhile, a squeeze-and-excitation feature fusion module (SEFM) is designed to merge two kinds of features via the weights of two kinds of features learning. In the squeeze step, appearance feature maps and pose feature maps are integrated via pooling, concatenation, and projection. In the excitation step, we obtain the weighted feature maps of appearance and pose via projection and Hadamard product. The two kinds of feature maps are down-sampled and concatenated into the final gait feature in accordance with adaptive weighting. To verify the appearance features and pose features, we design two variants of SEFM in related to SEFM-A and SEFM-P further. The SEFM module merges appearance features and pose features in mutual; the SEFM-A module merges pose features into appearance features and appearance features remain unchanged; the SEFM-P module merges appearance features into pose features and no pose features changed. Our algorithm is based on Pytorch and the evaluation is carried out on database CASIA (Institute of Automation, Chinese Academy of Sciences) Gait Dataset B (CASIA-B). We adopt the AlphaPose algorithm to extract pose key-points from origin RGB videos, and use silhouette images obtained. In each iteration of the training process, we randomly select 16 subjects and select 8 random samples of each subject further. Every sample of them contains a sub-sequence of 30 frames. Consequently, each batch has 3 840 image-skeleton pairs. We adopt the Adam optimizer to optimize the network for 60 000 iterations. The initial learning rate is set to 0. 000 2 for the pose branch, and 0. 000 1 for the appearance branch and the SEFM, and then the learning rate is cut10 times at the 45 000-th iteration. Result We first verify the effectiveness of the dual-branch network and feature fusion modules. Our demonstration illustrates that our dual-branch network can enhance performance and there is a clear complementary effect between appearance features and pose features. The Rank-1 accuracies of five feature fusion modules like SEFM, SEFM-A, SEFM-P, Concatenation, and multi-modal transfer module (MMTM) are 83. 5%, 81. 9%, 93. 4%, 92. 6% and 79. 5%, respectively. These results demonstrate that appearance features are more discriminative because there are noises existed in pose features. Our SEFM-P is capable to merge two features in the feature fusion procedure via noises suppression. Then, we compare our methods to advanced gait recognition methods like CNNs, event-based gait recognition (EV-Gait), GaitSet, and PoseGait. We conduct the experiments with two protocols and evaluate the rank-1 accuracy of three walking scenarios in the context of normal walking, bag-carrying, and coat-wearing. Our method archives the best performance in all experimental protocols. Our three scenarios-based rank-1 accuracies are reached 93. 4%, 84. 8%, and 70. 9% in protocol 1. The results of protocol 2 are obtained by 95. 7%, 87. 8%, 77. 0%, respectively. Comparing to the second-best method of GaitSet, the rank-1 accuracies in the context of coat-wearing walking scenario are improved by 8. 4% and 6. 6% . Conclusion We harness a novel gait recognition network based on the fusions of appearance features and pose features. Our analyzed results demonstrated that our method can develop two kinds of features and the appearance variations is more robust, especially for clothing changes scenario.

biometric recognition feature fusion gait recognition gait silhouette images human body pose estimation squeeze-and-excitation module two-branch network

中文

其他

Shanxi Provincial Key Research and Development Project[202004a07020050];National Natural Science Foundation of China[61772032];National Key Research and Development Program of China[SQ2018YFC080102];

2-s2.0-85137811517

Scopus

1.School of Electronics and Information Engineering,Anhui University,Hefei,230601,China
2.College of Engineering,Southern University of Science and Technology,Shenzhen,518055,China

Xu，Shuo,Zheng，Feng,Tang，Jun,等. Dual branch feature fusion network based gait recognition algorithm 双分支特征融合网络的步态识别算法[J]. 中国图象图形学报,2022,27(7):2263-2273.

Xu，Shuo,Zheng，Feng,Tang，Jun,&Bao，Wenxia.(2022).Dual branch feature fusion network based gait recognition algorithm 双分支特征融合网络的步态识别算法.中国图象图形学报,27(7),2263-2273.

Xu，Shuo,et al."Dual branch feature fusion network based gait recognition algorithm 双分支特征融合网络的步态识别算法".中国图象图形学报 27.7(2022):2263-2273.