Hybrid Graph Convolutional Networks for Skeleton-Based and EEG-Based Jumping Action Recognition

Feng, Naishi1; Hu, Fo1; Wang, Hong1; Zhao, Ziqi2

10.1109/IROS51168.2021.9636110

2021

IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

2153-0858

978-1-6654-1715-0

2021 IEEE/RSJ INTERNATIONAL CONFERENCE ON INTELLIGENT ROBOTS AND SYSTEMS (IROS)

4156-4161

SEP 27-OCT 01, 2021

null,null,ELECTR NETWORK

345 E 47TH ST, NEW YORK, NY 10017 USA

IEEE

Kinematic information obtained directly from the skeletal model has been useful for jumping action recognition. Current research focuses on dynamic analysis based on the video stream. Although skeletal data can accurately capture the high-level information of human action, it ignores the brain's pre-execution command information, which plays a crucial role in identifying jumping action. Therefore, we proposed a hybrid model based on brain network and dynamic skeleton. Specifically, we used a brain network graph convolutional network (BNGCN) to encode brain command information. Also, a dynamic skeleton convolutional network (DSGCN) using the angular velocity of skeleton nodes instead of video is proposed, which can break the fixed experimental area's limitation. BNGCN and DSGCN are fused through three network nodes to construct an end-to-end Brain Network and Dynamic Skeleton Hybrid Model. Our contribution consists of three parts. First, we have created a data set that can be used for jumping action and its sub-phase recognition. Second, BNGCN is used to extract brain command information for jumping action recognition. Third, a hybrid model is proposed to incorporate brain command and skeleton kinematic information. The results show that our hybrid model can effectively capture the high-level features for jumping action recognition. The method outperforms compared methods for jumping action recognition.

Fuses Kinematics Streaming media Brain modeling Feature extraction Skeleton Hybrid power systems

其他

英语

CPCI-S ; EI

Automation & Control Systems ; Computer Science ; Engineering ; Robotics

Automation & Control Systems ; Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic ; Robotics

WOS:000755125503039

20220711623842

Convolutional neural networks ; Kinematics ; Musculoskeletal system

Biomechanics, Bionics and Biomimetics:461.3 ; Information Theory and Signal Processing:716.1 ; Mechanics:931.1

Web of Science

1.Northeastern Univ, Dept Mech Engn & Automat, Shenyang 110819, Peoples R China
2.Southern Univ Sci & Technol, Dept Elect & Elect Engn, Shenzhen, Peoples R China

Feng, Naishi,Hu, Fo,Wang, Hong,et al. Hybrid Graph Convolutional Networks for Skeleton-Based and EEG-Based Jumping Action Recognition[C]. 345 E 47TH ST, NEW YORK, NY 10017 USA:IEEE,2021:4156-4161.