Multi-modal Segment Assemblage Network for Ad Video Editing with Importance-Coherence Reward

Tang，Yunlong1,2; Xu，Siting1; Wang，Teng1,2; Lin，Qin2; Lu，Qinglin2; Zheng，Feng1

10.1007/978-3-031-26284-5_34

2023

0302-9743

1611-3349

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

13842 LNCS

560-576

Advertisement video editing aims to automatically edit advertising videos into shorter videos while retaining coherent content and crucial information conveyed by advertisers. It mainly contains two stages: video segmentation and segment assemblage. The existing method performs well at video segmentation stages but suffers from the problems of dependencies on extra cumbersome models and poor performance at the segment assemblage stage. To address these problems, we propose M-SAN (Multi-modal Segment Assemblage Network) which can perform efficient and coherent segment assemblage task end-to-end. It utilizes multi-modal representation extracted from the segments and follows the Encoder-Decoder Ptr-Net framework with the Attention mechanism. Importance-coherence reward is designed for training M-SAN. We experiment on the Ads-1k dataset with 1000+ videos under rich ad scenarios collected from advertisers. To evaluate the methods, we propose a unified metric, Imp-Coh@Time, which comprehensively assesses the importance, coherence, and duration of the outputs at the same time. Experimental results show that our method achieves better performance than random selection and the previous method on the metric. Ablation experiments further verify that multi-modal representation and importance-coherence reward significantly improve the performance. Ads-1k dataset is available at: https://github.com/yunlong10/Ads-1k.

Ad video editing Advertisement dataset Multi-modal Segment assemblage Video segmentation Video summarization

第一 ; 通讯

英语

EI

20231113699436

Image segmentation ; Video recording ; Videodisks

Television Systems and Equipment:716.4 ; Optical Devices and Systems:741.3

2-s2.0-85149633868

Scopus

1.Southern University of Science and Technology,Shenzhen,China
2.Tencent Inc.,Shenzhen,China

Tang，Yunlong,Xu，Siting,Wang，Teng,et al. Multi-modal Segment Assemblage Network for Ad Video Editing with Importance-Coherence Reward[C],2023:560-576.