Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models

Dong Lu1; Zhiqiang Wang1; Teng Wang1,2; Weili Guan3; Hongchang Gao4; Feng Zheng1,5

10.1109/ICCV51070.2023.00016

2023-10-02

2023ICCV

1550-5499

979-8-3503-0719-1

2023 IEEE/CVF International Conference on Computer Vision (ICCV)

102-111

2023-10-2~10-6

法国巴黎

10662 LOS VAQUEROS CIRCLE, PO BOX 3014, LOS ALAMITOS, CA 90720-1264 USA

IEEE COMPUTER SOC

Vision-language pre-training (VLP) models have shown vulnerability to adversarial examples in multimodal tasks. Furthermore, malicious adversaries can be deliberately transferred to attack other black-box models. However, existing work has mainly focused on investigating white-box attacks. In this paper, we present the first study to investigate the adversarial transferability of recent VLP models. We observe that existing methods exhibit much lower transferability, compared to the strong attack performance in white-box settings. The transferability degradation is partly caused by the under-utilization of cross-modal interactions. Particularly, unlike unimodal learning, VLP models rely heavily on cross-modal interactions and the multimodal alignments are many-to-many, e.g., an image can be described in various natural languages. To this end, we propose a highly transferable Set-level Guidance Attack (SGA) that thoroughly leverages modality interactions and incorporates alignment-preserving augmentation with cross-modal guidance. Experimental results demonstrate that SGA could generate adversarial examples that can strongly transfer across different VLP models on multiple downstream vision-language tasks. On image-text retrieval, SGA significantly enhances the attack success rate for transfer attacks from ALBEF to TCL by a large margin (at least 9.78% and up to 30.21%), compared to the state-of-the-art. Our code is available at https://github.com/Zoky-2020/SGA.

Degradation Computer vision Codes Computational modeling Natural languages Closed box Boosting

第一 ; 共同第一 ; 通讯

英语

CPCI-S

National Key R&D Program of China[2022YFF1202903] ; National Natural Science Foundation of China[62122035]

Computer Science ; Imaging Science & Photographic Technology

Computer Science, Artificial Intelligence ; Computer Science, Theory & Methods ; Imaging Science & Photographic Technology

WOS:001159644300010

人工提交

1.Southern University of Science and Technology
2.The University of Hong Kong
3.Monash University
4.Temple University
5.Peng Cheng Laboratory

Dong Lu,Zhiqiang Wang,Teng Wang,et al. Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models[C]. 10662 LOS VAQUEROS CIRCLE, PO BOX 3014, LOS ALAMITOS, CA 90720-1264 USA:IEEE COMPUTER SOC,2023:102-111.