Learning Cross-Modal Affinity for Referring Video Object Segmentation Targeting Limited Samples

Guanghui Li1; Mingqi Gao2; Heng Liu1; Xiantong Zhen3; Feng Zheng2

10.1109/ICCV51070.2023.00253

2023

IEEE/CVF International Conference on Computer Vision (ICCV)

1550-5499

979-8-3503-0719-1

2023 IEEE/CVF International Conference on Computer Vision (ICCV)

2684-2693

1-6 Oct. 2023

Paris, France

10662 LOS VAQUEROS CIRCLE, PO BOX 3014, LOS ALAMITOS, CA 90720-1264 USA

IEEE COMPUTER SOC

Referring video object segmentation (RVOS), as a supervised learning task, relies on sufficient annotated data for a given scene. However, in more realistic scenarios, only minimal annotations are available for a new scene, which poses significant challenges to existing RVOS methods. With this in mind, we propose a simple yet effective model with a newly designed cross-modal affinity (CMA) module based on a Transformer architecture. The CMA module builds multimodal affinity with a few samples, thus quickly learning new semantic information, and enabling the model to adapt to different scenarios. Since the proposed method targets limited samples for new scenes, we generalize the problem as - few-shot referring video object segmentation (FS-RVOS). To foster research in this direction, we build up a new FS-RVOS benchmark based on currently available datasets. The benchmark covers a wide range and includes multiple situations, which can maximally simulate real-world scenarios. Extensive experiments show that our model adapts well to different scenarios with only a few samples, reaching state-of-the-art performance on the benchmark. On Mini-Ref-YouTube-VOS, our model achieves an average performance of 53.1 and 54.8, which are 10% better than the baselines. Furthermore, we show impressive results of 77.7 and 74.8 on Mini-Ref-SAIL-VOS, which are significantly better than the baselines. Code is publicly available at https://github.com/hengliusky/Few_shot_RVOS.

Adaptation models Computer vision Computational modeling Supervised learning Semantics Object segmentation Computer architecture

其他

英语

CPCI-S

National Key R&D Program of China[2022YFF1202903] ; National Natural Science Foundation of China["61971004","62122035"] ; Natural Science Foundation of Anhui Province, China[2008085MF190] ; Equipment Advanced Research Sharing Technology Project, China[80912020104]

Computer Science ; Imaging Science & Photographic Technology

Computer Science, Artificial Intelligence ; Computer Science, Theory & Methods ; Imaging Science & Photographic Technology

WOS:001159644302087

IEEE

1.Anhui University of Technology
2.Southern University of Science and Technology
3.United Imaging

Guanghui Li,Mingqi Gao,Heng Liu,et al. Learning Cross-Modal Affinity for Referring Video Object Segmentation Targeting Limited Samples[C]. 10662 LOS VAQUEROS CIRCLE, PO BOX 3014, LOS ALAMITOS, CA 90720-1264 USA:IEEE COMPUTER SOC,2023:2684-2693.