Leveraging per Image-Token Consistency for Vision-Language Pre-training

Yunhao Gou1; Tom Ko2; Hansi Yang3; James Kwok3; Yu Zhang1; Mingxuan Wang2

10.1109/CVPR52729.2023.01836

2023

1063-6919

979-8-3503-0130-4

2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

19155-19164

17-24 June 2023

Vancouver, BC, Canada

Most existing vision-language pre-training (VLP) approaches adopt cross-modal masked language modeling (CMLM) to learn vision-language associations. However, we find that CMLM is insufficient for this purpose according to our observations: (1) Modality bias: a considerable amount of masked tokens in CMLM can be recovered with only the language information, ignoring the visual inputs. (2) Underutilization of the unmasked tokens: CMLM primarily focuses on the masked token but it cannot simultaneously leverage other tokens to learn vision-language associations. To handle those limitations, we propose EPIC (lEveraging Per Image-Token Consistency for vision-language pre-training). In EPIC, for each image-sentence pair, we mask tokens that are salient to the image (i.e., Saliency-based Masking Strategy) and replace them with alternatives sampled from a language model (i.e., Inconsistent Token Generation Procedure), and then the model is required to determine for each token in the sentence whether it is consistent with the image (i.e., Image-Token Consistency Task). The proposed EPIC method is easily combined with pre-training methods. Extensive experiments show that the combination of the EPIC method and state-of-the-art pre-training approaches, including ViLT, ALBEF, METER, and X-VLM, leads to significant improvements on downstream tasks. Our coude is released at https://github.com/gyhdog99/epic

Multi-modal learning

第一

CPCI-S

WOS:001062531303045

IEEE

1.Southern University of Science and Technology
2.ByteDance AI Lab
3.Hong Kong University of Science and Technology

Yunhao Gou,Tom Ko,Hansi Yang,et al. Leveraging per Image-Token Consistency for Vision-Language Pre-training[C],2023:19155-19164.