Transferable Decoding with Visual Entities for Zero-Shot Image Captioning

Junjie Fei1; Teng Wang1,2; Jinrui Zhang1; Zhenyu He3; Chengjie Wang44,5; Feng Zheng1

10.48550/arXiv.2307.16525

2023-10-02

2023ICCV

1550-5499

979-8-3503-0719-1

2023 IEEE/CVF International Conference on Computer Vision (ICCV)

3113-3123

2023-10-2~10-6

法国巴黎

10662 LOS VAQUEROS CIRCLE, PO BOX 3014, LOS ALAMITOS, CA 90720-1264 USA

IEEE COMPUTER SOC

Image-to-text generation aims to describe images using natural language. Recently, zero-shot image captioning based on pre-trained vision-language models (VLMs) and large language models (LLMs) has made significant progress. However, we have observed and empirically demonstrated that these methods are susceptible to modality bias induced by LLMs and tend to generate descriptions containing objects (entities) that do not actually exist in the image but frequently appear during training (i.e., object hallucination). In this paper, we propose ViECap, a transferable decoding model that leverages entity-aware decoding to generate descriptions in both seen and unseen scenarios. ViECap incorporates entity-aware hard prompts to guide LLMs' attention toward the visual entities present in the image, enabling coherent caption generation across diverse scenes. With entity-aware hard prompts, ViECap is capable of maintaining performance when transferring from in-domain to out-of-domain scenarios. Extensive experiments demonstrate that ViECap sets a new state-of-theart cross-domain (transferable) captioning and performs competitively in-domain captioning compared to previous VLMs-based zero- shot methods. Our code is available at: https://github.com/FeiElysia/ViECap

Training Visualization Computer vision Adaptation models Codes Computational modeling Natural languages

第一 ; 共同第一 ; 通讯

英语

CPCI-S

National Key R&D Program of China[2022YFF1202903] ; National Natural Science Foundation of China[62122035]

Computer Science ; Imaging Science & Photographic Technology

Computer Science, Artificial Intelligence ; Computer Science, Theory & Methods ; Imaging Science & Photographic Technology

WOS:001159644303034

人工提交

1.Southern University of Science and Technology
2.The University of Hong Kong
3.Harbin Institute of Technology (Shenzhen)
4.Tencent
5.Shanghai Jiao Tong University

Junjie Fei,Teng Wang,Jinrui Zhang,et al. Transferable Decoding with Visual Entities for Zero-Shot Image Captioning[C]. 10662 LOS VAQUEROS CIRCLE, PO BOX 3014, LOS ALAMITOS, CA 90720-1264 USA:IEEE COMPUTER SOC,2023:3113-3123.