Open-set Synthesis for Free-viewpoint Human Body Reenactment of Novel Poses

Zhicheng Sheng1; Fan Liu2; Meng Liu3; Feng Zheng4; Liqiang Nie5

2024

10.1109/TCSVT.2024.3434733

IEEE Transactions on Circuits and Systems for Video Technology   影响因子和分区

1558-2205

Free-viewpoint human body reenactment aims to generate authentic and coherent poses for a source subject based on a target body pose skeleton. While current methods are proficient in reproducing existing poses, they falter in generating novel poses, often yielding blurry results. In this paper, we propose a method for open-set pose synthesis, utilizing multi-view images to generate novel poses from arbitrary viewpoints. Our method begins with building the neural radiance fields (NeRF) using multi-view images. While this NeRF is adept at rendering specific free-viewpoint refined poses, it struggles with sharp results for novel poses. To address this, we introduce a 2D novel pose diffusion (2D-NPD) module and a view-consistent NeRF optimization (VCNeRF-O) strategy. The 2D-NPD performs body reenactment in the 2D domain to generate a set of refined novel pose images. In particular, we introduce a motion adapter tailored for the stable diffusion (SD) model to generate novel poses while preserving the cloth texture. To ensure seamless motion and image clarity, we further devise a dual warp loss function for the motion adapter. Moreover, to generate fine-grained novel poses while maintaining viewpoint consistency, we develop an innovative VCNeRF-O to optimize the NeRF. Experiments demonstrate that our approach outperforms existing techniques in terms of texture quality and consistency in the open-set synthesis of novel poses.

EI

其他

ENGINEERING

1.School of Computer Science and Technology, Shandong University, Qingdao, China
2.School of Computing, National University of Singapore, Singapore
3.School of Computer Science and Technology, Shandong Jianzhu University, Jinan, China
4.Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China
5.School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen), Shenzhen, China

Zhicheng Sheng,Fan Liu,Meng Liu,et al. Open-set Synthesis for Free-viewpoint Human Body Reenactment of Novel Poses[J]. IEEE Transactions on Circuits and Systems for Video Technology,2024,PP(99).

Zhicheng Sheng,Fan Liu,Meng Liu,Feng Zheng,&Liqiang Nie.(2024).Open-set Synthesis for Free-viewpoint Human Body Reenactment of Novel Poses.IEEE Transactions on Circuits and Systems for Video Technology,PP(99).

Zhicheng Sheng,et al."Open-set Synthesis for Free-viewpoint Human Body Reenactment of Novel Poses".IEEE Transactions on Circuits and Systems for Video Technology PP.99(2024).