Bidirectional Error-Aware Fusion Network for Video Inpainting

Jiacheng Hou1; Zhong Ji1; Jinyu Yang2,3; Feng Zheng2

2024

10.1109/TCSVT.2024.3454641

IEEE Transactions on Circuits and Systems for Video Technology   影响因子和分区

1558-2205

Existing video inpainting approaches tend to adopt vision transformers with rare customized designs, which poses two limitations. Firstly, the conventional self-attention mechanism treats tokens from invalid and valid regions equally and mingles them, which may incur blurriness. Secondly, these approaches merely employ forward frames as references, while ignoring the past inpainted frames, which are also valuable in enhancing temporal consistency and offering more available information. In this paper, we propose a new video inpainting network, called Bidirectional Error-Aware Fusion Network (BEAF-Net). Concretely, on one hand, we propose a tailored Error-Aware Transformer (EAT) that discerns different tokens by assigning dynamic weights to bridle the use of erroneous tokens. Meanwhile, each EAT is equipped with a Spatial Feature Enhancement (SFE) layer to synthesize features with multi-scales. On the other hand, we apply a pair of EATs to utilize forward reference frames and past inpainted frames simultaneously, and a proposed Bidirectional Fusion (BiF) layer is exerted to blend the aggregation results adaptively. By coupling these novel designs, our proposed BEAF-Net completely leverages the location priors, multi-scale perception, and past predictions to produce more faithful and consistent inpainting results. We corroborate our BEAF-Net on two commonly-used video inpainting datasets: DAVIS and Youtube-VOS, where the experimental results demonstrate BEAF-Net compares favorably with state-of-the-art solutions. Video examples can be found at https://github.com/JCATCV/BEAF-Net.

其他

1.School of Electrical and Information Engineering; Tianjin Key Laboratory of Brain-inspired Intelligence Technology, Tianjin University, Tianjin, China
2.Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China
3.School of Computer Science, University of Birmingham, Birmingham, U.K.

Jiacheng Hou,Zhong Ji,Jinyu Yang,et al. Bidirectional Error-Aware Fusion Network for Video Inpainting[J]. IEEE Transactions on Circuits and Systems for Video Technology,2024,PP(99).

Jiacheng Hou,Zhong Ji,Jinyu Yang,&Feng Zheng.(2024).Bidirectional Error-Aware Fusion Network for Video Inpainting.IEEE Transactions on Circuits and Systems for Video Technology,PP(99).

Jiacheng Hou,et al."Bidirectional Error-Aware Fusion Network for Video Inpainting".IEEE Transactions on Circuits and Systems for Video Technology PP.99(2024).