Multi-modal recursive prompt learning with mixup embedding for generalization recognition

Jia，Yunpeng1; Ye，Xiufen1; Liu，Yusong2; Guo，Shuxiang3

2024-06-21

10.1016/j.knosys.2024.111726

Knowledge-Based Systems   影响因子和分区

0950-7051

The contrastive language-image pretraining (CLIP) model has shown promise in generalization recognition by combining visual and textual embeddings. However, the pretrained CLIP model requires further fine tuning for downstream tasks. Existing prompt learning (PL) approaches, while effective, neglect the cross-hierarchical fusion of multimodal features. To address this issue, we introduce multi-modal recursive PL (MmRPL) for different generalization image recognition tasks. Recursive connections between prompts at different layers facilitate hierarchy-aware vision-text fusion. To our best knowledge, we introduce a mixup embedding technique to enhance feature representations of PL for the first time. Extensive experiments across popular generalization recognition settings reveal that MmRPL outperforms existing methods. Furthermore, when combined with the embedding technique, MmRPL further enhances recognition in base-to-novel generalization, generalized zero-shot, domain generalization, and domain adaptive zero-shot scenarios. Notably, the proposed methods extend to applications like zero-shot recognition of side-scan sonar (SSS) image targets via domain generalization adaptation. The proposed methods also achieve more than 80% mean accuracy among seafloor, airplane, and ship categories in SSS images.

Domain adaptation Generalization recognition Mixup technique Prompt learning

SCI ; EI

英语

其他

COMPUTER SCIENCE

2-s2.0-85189756837

Scopus

1.College of Intelligent Systems Science and Engineering,Harbin Engineering University,Harbin,150001,China
2.School of Interdisciplinary Medicine and Engineering,Harbin Medical University,Harbin,150081,China
3.Department of Electronic and Electrical Engineering,Southern University of Science and Technology,Shenzhen,518055,China

Jia，Yunpeng,Ye，Xiufen,Liu，Yusong,et al. Multi-modal recursive prompt learning with mixup embedding for generalization recognition[J]. Knowledge-Based Systems,2024,294.

Jia，Yunpeng,Ye，Xiufen,Liu，Yusong,&Guo，Shuxiang.(2024).Multi-modal recursive prompt learning with mixup embedding for generalization recognition.Knowledge-Based Systems,294.

Jia，Yunpeng,et al."Multi-modal recursive prompt learning with mixup embedding for generalization recognition".Knowledge-Based Systems 294(2024).