WeClick: Weakly-Supervised Video Semantic Segmentation with Click Annotations

Liu，Peidong1; He，Zibin1; Yan，Xiyu1; Jiang，Yong1,2; Xia，Shu Tao1,2; Zheng，Feng3; Maowei，Hu1,4

10.1145/3474085.3475217

2021-10-17

MM 2021 - Proceedings of the 29th ACM International Conference on Multimedia

2995-3004

Compared with tedious per-pixel mask annotating, it is much easier to annotate data by clicks, which costs only several seconds for an image. However, applying clicks to learn video semantic segmentation model has not been explored before. In this work, we propose an effective weakly-supervised video semantic segmentation pipeline with click annotations, called WeClick, for saving laborious annotating effort by segmenting an instance of the semantic class with only a single click. Since detailed semantic information is not captured by clicks, directly training with click labels leads to poor segmentation predictions. To mitigate this problem, we design a novel memory flow knowledge distillation strategy to exploit temporal information (named memory flow) in abundant unlabeled video frames, by distilling the neighboring predictions to the target frame via estimated motion. Moreover, we adopt vanilla knowledge distillation for model compression. In this case, WeClick learns compact video semantic segmentation models with the low-cost click annotations during the training phase yet achieves real-time and accurate models during the inference period. Experimental results on Cityscapes and Camvid show that WeClick outperforms the state-of-the-art methods, increases performance by 10.24% mIoU than baseline, and achieves real-time execution.

click annotations knowledge distillation video semantic segmentation weakly-supervised learning

其他

英语

EI

20214711200266

Computer vision ; Distillation ; Semantics

Artificial Intelligence:723.4 ; Computer Applications:723.5 ; Vision:741.2 ; Chemical Operations:802.3

2-s2.0-85119357398

Scopus

1.Tsinghua Shenzhen International Graduate School,Tsinghua University,China
2.PCL Research Center of Networks and Communications,Peng Cheng Laboratory,China
3.Department of Computer Science and Engineering,Southern University of Science and Technology,China
4.Shenzhen Rejoice Sport Tech. Co.,LTD,China

Liu，Peidong,He，Zibin,Yan，Xiyu,et al. WeClick: Weakly-Supervised Video Semantic Segmentation with Click Annotations[C],2021:2995-3004.