Dense-Localizing Audio-Visual Events in Untrimmed Videos: A Large-Scale Benchmark and Baseline

Tiantian Geng1; Teng Wang1; Jinming Duan2; Runmin Cong3; Feng Zheng1

10.1109/CVPR52729.2023.02197

2023

1063-6919

979-8-3503-0130-4

2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

22942-22951

17-24 June 2023

Vancouver, BC, Canada

Existing audio-visual event localization (AVE) handles manually trimmed videos with only a single instance in each of them. However, this setting is unrealistic as natural videos often contain numerous audio-visual events with different categories. To better adapt to real-life applications, in this paper we focus on the task of dense-localizing audio-visual events, which aims to jointly localize and recognize all audio-visual events occurring in an untrimmed video. The problem is challenging as it requires fine-grained audio-visual scene and context understanding. To tackle this problem, we introduce the first Untrimmed Audio-Visual (UnAV-J 00) dataset, which contains 10K untrimmed videos with over 30K audio-visual events. Each video has 2.8 audio-visual events on average, and the events are usually related to each other and might co-occur as in real-life scenes. Next, we formulate the task using a new learning-based framework, which is capable of fully integrating audio and visual modalities to localize audio-visual events with various lengths and capture dependencies between them in a single pass. Extensive experiments demonstrate the effectiveness of our method as well as the significance of multi-scale cross-modal perception and dependency modeling for this task. The dataset and code are available at https://unav100.github.io.

Video: Action and event understanding

第一

CPCI-S

WOS:001062531307026

IEEE

1.Southern University of Science and Technology
2.University of Birmingham
3.Shandong University

Tiantian Geng,Teng Wang,Jinming Duan,et al. Dense-Localizing Audio-Visual Events in Untrimmed Videos: A Large-Scale Benchmark and Baseline[C],2023:22942-22951.