InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referring

Yuan，Zhihao1; Yan，Xu1; Liao，Yinghong1; Zhang，Ruimao1; Wang，Sheng2; Li，Zhen1; Cui，Shuguang1

10.1109/ICCV48922.2021.00181

2021

1550-5499

978-1-6654-2813-2

Proceedings of the IEEE International Conference on Computer Vision

1771-1780

10-17 Oct. 2021

Montreal, QC, Canada

Compared with the visual grounding on 2D images, the natural-language-guided 3D object localization on point clouds is more challenging. In this paper, we propose a new model, named InstanceRefer, to achieve a superior 3D visual grounding through the grounding-by-matching strategy. In practice, our model first predicts the target category from the language descriptions using a simple language classification model. Then, based on the category, our model sifts out a small number of instance candidates (usually less than 20) from the panoptic segmentation on point clouds. Thus, the non-trivial 3D visual grounding task has been effectively re-formulated as a simplified instance-matching problem, considering that instance-level candidates are more rational than the redundant 3D object proposals. Subsequently, for each candidate, we perform the multi-level contextual inference, i.e., referring from instance attribute perception, instance-to-instance relation perception, and instance-to-background global localization perception, respectively. Eventually, the most relevant candidate is selected and localized by ranking confidence scores, which are obtained by the cooperative holistic visual-language feature matching. Experiments confirm that our method outperforms previous state-of-the-arts on ScanRefer online benchmark and Nr3D/Sr3D datasets.

Vision + language Detection and localization in 2D and 3D Scene analysis and understanding Visual reasoning and logical representation

其他

英语

EI

National Key Research and Development Program of China[2018YFB1800800];

20221511951300

2-s2.0-85120974687

Scopus

1.The Chinese University of Hong Kong (Shenzhen),Shenzhen Research Institute of Big Data,Hong Kong
2.CryoEM Center,Southern University of Science and Technology,China

Yuan，Zhihao,Yan，Xu,Liao，Yinghong,et al. InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referring[C],2021:1771-1780.