Memory-attended recurrent network for video captioning

Pei, Wenjie1; Zhang, Jiyuan1; Wang, Xiangrong2; Ke, Lei1; Shen, Xiaoyong1; Tai, Yu-Wing1

10.1109/CVPR.2019.00854

2019

1063-6919

978-1-7281-3294-5

32nd IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2019

2019-June

8339-8348

15-20 June 2019

Long Beach, CA, United states

345 E 47TH ST, NEW YORK, NY 10017 USA

IEEE Computer Society

Typical techniques for video captioning follow the encoder-decoder framework, which can only focus on one source video being processed. A potential disadvantage of such design is that it cannot capture the multiple visual context information of a word appearing in more than one relevant videos in training data. To tackle this limitation, we propose the Memory-Attended Recurrent Network (MARN) for video captioning, in which a memory structure is designed to explore the full-spectrum correspondence between a word and its various similar visual contexts across videos in training data. Thus, our model is able to achieve a more comprehensive understanding for each word and yield higher captioning quality. Furthermore, the built memory structure enables our method to model the compatibility between adjacent words explicitly instead of asking the model to learn implicitly, as most existing models do. Extensive validation on two real-word datasets demonstrates that our MARN consistently outperforms state-of-the-art methods.
© 2019 IEEE.

Vision + Language Deep Learning Video Analytics

其他

英语

EI ; CPCI-S

Computer Science

Computer Science, Artificial Intelligence ; Computer Science, Theory & Methods

WOS:000542649301097

20200508114468

Recurrent neural networks

Computer Applications:723.5 ; Vision:741.2

EV Compendex

1.Tencent, China
2.Southern University of Science and Technology, China

Pei, Wenjie,Zhang, Jiyuan,Wang, Xiangrong,et al. Memory-attended recurrent network for video captioning[C]. 345 E 47TH ST, NEW YORK, NY 10017 USA:IEEE Computer Society,2019:8339-8348.