Fast parameter adaptation for few-shot image captioning and visual question answering

Dong, Xuanyi1,2; Zhu, Linchao1,2; Zhang, De3; Yang, Yi1,2; Wu, Fei4

10.1145/3240508.3240527

2018

26th ACM Multimedia conference, MM 2018

54-62

Seoul, Korea, Republic of

1515 BROADWAY, NEW YORK, NY 10036-9998 USA

Association for Computing Machinery, Inc

Given only a few image-text pairs, humans can learn to detect semantic concepts and describe the content. For machine learning algorithms, they usually require a lot of data to train a deep neural network to solve the problem. However, it is challenging for the existing systems to generalize well to the few-shot multi-modal scenario, because the learner should understand not only images and texts but also their relationships from only a few examples. In this paper, we tackle two multi-modal problems, i.e., image captioning and visual question answering (VQA), in the few-shot setting. We propose Fast Parameter Adaptation for Image-Text Modeling (FPAIT) that learns to learn jointly understanding image and text data by a few examples. In practice, FPAIT has two benefits. (1) Fast learning ability. FPAIT learns proper initial parameters for the joint image-text learner from a large number of different tasks. When a new task comes, FPAIT can use a small number of gradient steps to achieve a good performance. (2) Robust to few examples. In few-shot tasks, the small training data will introduce large biases in Convolutional Neural Networks (CNN) and damage the learner's performance. FPAIT leverages dynamic linear transformations to alleviate the side effects of the small training set. In this way, FPAIT flexibly normalizes the features and thus reduces the biases during training. Quantitatively, FPAIT achieves superior performance on both few-shot image captioning and VQA benchmarks.
© 2018 Association for Computing Machinery.

Few-shot Learning Image Captioning Visual Question Answering

第一 ; 通讯

英语

EI ; CPCI-S

Data to Decisions Cooperative Research Centres[]

Computer Science ; Engineering

Computer Science, Theory & Methods ; Engineering, Electrical & Electronic

WOS:000509665700007

20185006246260

Benchmarking ; Deep neural networks ; Linear transformations ; Mathematical transformations ; Natural language processing systems ; Neural networks ; Semantics

Data Processing and Image Processing:723.2 ; Mathematical Transformations:921.3

EV Compendex

1.SUSTech-UTS Joint Centre of CIS, Southern University of Science and Technology, United States
2.CAI, University of Technology Sydney, Australia
3.Information Science Academy, CETC, China
4.CCST, Zhejiang University, China

Dong, Xuanyi,Zhu, Linchao,Zhang, De,et al. Fast parameter adaptation for few-shot image captioning and visual question answering[C]. 1515 BROADWAY, NEW YORK, NY 10036-9998 USA:Association for Computing Machinery, Inc,2018:54-62.