Knowledge-Aware Prompt Tuning for Generalizable Vision-Language Models

Baoshuo Kan1; Teng Wang2,3; Wenpeng Lu1; Xiantong Zhen4; Weili Guan5; Feng Zheng2

10.1109/ICCV51070.2023.01436

2023-10-02

2023ICCV

1550-5499

979-8-3503-0719-1

2023 IEEE/CVF International Conference on Computer Vision (ICCV)

15624-15634

2023-10-2~10-6

法国巴黎

10662 LOS VAQUEROS CIRCLE, PO BOX 3014, LOS ALAMITOS, CA 90720-1264 USA

IEEE COMPUTER SOC

Pre-trained vision-language models, e.g., CLIP, working with manually designed prompts have demonstrated great capacity of transfer learning. Recently, learnable prompts achieve state-of-the-art performance, which however are prone to overfit to seen classes, failing to generalize to unseen classes. In this paper, we propose a Knowledge-Aware Prompt Tuning (KAPT) framework for vision-language models. Our approach takes the inspiration from human intelligence in which external knowledge is usually incorporated into recognizing novel categories of objects. Specifically, we design two complementary types of knowledge-aware prompts for the text encoder to leverage the distinctive characteristics of category-related external knowledge. The discrete prompt extracts the key information from descriptions of an object category, and the learned continuous prompt captures overall contexts. We further design an adaptation head for the visual encoder to aggregate salient attentive visual cues, which establishes discriminative and task-aware visual representations. We conduct extensive experiments on 11 widely-used benchmark datasets and the results verify the effectiveness in fewshot image classification, especially in generalizing to unseen categories. Compared with the state-of-the-art CoCoOp method, KAPT exhibits favorable performance and achieves an absolute gain of 3.22% on new classes and 2.57% in terms of harmonic mean.

Visualization Adaptation models Head Knowledge based systems Transfer learning Encyclopedias Benchmark testing

共同第一 ; 通讯

英语

CPCI-S

National Key R&D Program of China[2022YFF1202903]

Computer Science ; Imaging Science & Photographic Technology

Computer Science, Artificial Intelligence ; Computer Science, Theory & Methods ; Imaging Science & Photographic Technology

WOS:001169499008011

人工提交

1.Qilu University of Technology (Shandong Academy of Sciences)
2.Southern University of Science and Technology
3.The University of Hong Kong
4.United Imaging Healthcare
5.Monash Univerisity

Baoshuo Kan,Teng Wang,Wenpeng Lu,et al. Knowledge-Aware Prompt Tuning for Generalizable Vision-Language Models[C]. 10662 LOS VAQUEROS CIRCLE, PO BOX 3014, LOS ALAMITOS, CA 90720-1264 USA:IEEE COMPUTER SOC,2023:15624-15634.