Skating-Mixer: Long-Term Sport Audio-Visual Modeling with MLPs

Xia，Jingfei1,2; Zhuge，Mingchen1,3; Geng，Tiantian1; Fan，Shun1; Wei，Yuantai1; He，Zhenyu4; Zheng，Feng1

2023-06-27

37th AAAI Conference on Artificial Intelligence (AAAI) / 35th Conference on Innovative Applications of Artificial Intelligence / 13th Symposium on Educational Advances in Artificial Intelligence

2159-5399

2374-3468

*****************

Proceedings of the 37th AAAI Conference on Artificial Intelligence, AAAI 2023

37

2901-2909

FEB 07-14, 2023

null,Washington,DC

2275 E BAYSHORE RD, STE 160, PALO ALTO, CA 94303 USA

ASSOC ADVANCEMENT ARTIFICIAL INTELLIGENCE

Figure skating scoring is challenging because it requires judging the technical moves of the players as well as their coordination with the background music. Most learning-based methods cannot solve it well for two reasons: 1) each move in figure skating changes quickly, hence simply applying traditional frame sampling will lose a lot of valuable information, especially in 3 to 5 minutes long videos; 2) prior methods rarely considered the critical audio-visual relationship in their models. Due to these reasons, we introduce a novel architecture, named Skating-Mixer. It extends the MLP framework in a multimodal fashion and effectively learns longterm representations through our designed memory recurrent unit (MRU). Aside from the model, we collected a highquality audio-visual FS 1000 dataset, which contains over 1000 videos on 8 types of programs with 7 different rating metrics, overtaking other datasets in both quantity and diversity. Experiments show the proposed method achieves SOTAs over all major metrics on the public Fis-V and our FS 1000 dataset. In addition, we include an analysis applying our method to the recent competitions in Beijing 2022 Winter Olympic Games, proving our method has strong applicability.

第一 ; 通讯

英语

EI ; CPCI-S

National Natural Science Foundation of China[61972188];National Natural Science Foundation of China[62122035];

Computer Science

Computer Science, Artificial Intelligence ; Computer Science, Interdisciplinary Applications ; Computer Science, Theory & Methods

WOS:001243762100029

20233414583431

Artificial intelligence ; Mixers (machinery)

Artificial Intelligence:723.4

2-s2.0-85167997687

Scopus

1.Southern University of Science and Technology,China
2.The Chinese University of Hong Kong,Hong Kong
3.AI Initiative,King Abdullah University of Science and Technology (KAUST),Saudi Arabia
4.Harbin Institute of Technology (Shenzhen),China

Xia，Jingfei,Zhuge，Mingchen,Geng，Tiantian,et al. Skating-Mixer: Long-Term Sport Audio-Visual Modeling with MLPs[C]. 2275 E BAYSHORE RD, STE 160, PALO ALTO, CA 94303 USA:ASSOC ADVANCEMENT ARTIFICIAL INTELLIGENCE,2023:2901-2909.