Overcoming Data Deficiency for Multi-Person Pose Estimation

Dai, Yan1,2; Wang, Xuanhan1,2; Gao, Lianli1,2; Song, Jingkuan3; Zheng, Feng4; Shen, Heng Tao1,2

2023-05-01

10.1109/TNNLS.2023.3244957

IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS   影响因子和分区

2162-237X

2162-2388

Building multi-person pose estimation (MPPE) models that can handle complex foreground and uncommon scenes is an important challenge in computer vision. Aside from designing novel models, strengthening training data is a promising direction but remains largely unexploited for the MPPE task. In this article, we systematically identify the key deficiencies of existing pose datasets that prevent the power of well-designed models from being fully exploited and propose the corresponding solutions. Specifically, we find that the traditional data augmentation techniques are inadequate in addressing the two key deficiencies, imbalanced instance complexity (IC) (evaluated by our new metric IC) and insufficient realistic scenes. To overcome these deficiencies, we propose a model-agnostic full-view data generation (Full-DG) method to enrich the training data from the perspectives of both poses and scenes. By hallucinating images with more balanced pose complexity and richer real-world scenes, Full-DG can help improve pose estimators' robustness and generalizability. In addition, we introduce a plug-and-play adaptive category-aware loss (AC-loss) to alleviate the severe pixel-level imbalance between keypoints and backgrounds (i.e., around 1:600). Full-DG together with AC-loss can be readily applied to both the bottom-up and top-down models to improve their accuracy. Notably, plugging into the representative estimators HigherHRNet and HRNet, our method achieves substantial performance gains of 1.0%-2.9% AP on the COCO benchmark, and 1.0%-5.1% AP on the CrowdPose benchmark.

Pose estimation Task analysis Training Integrated circuit modeling Complexity theory Measurement Data models Data deficiency data generation fore-background imbalance multi-person pose estimation

SCI ; EI

英语

其他

National Key Research and Development Program of China["2022YFC2009903","2022YFC2009900"] ; National Natural Science Foundation of China["62122018","62020106008","61772116","61872064"] ; Fok Ying-Tong Education Foundation[171106] ; SongShan Laboratory[YYJC012022019]

Computer Science ; Engineering

Computer Science, Artificial Intelligence ; Computer Science, Hardware & Architecture ; Computer Science, Theory & Methods ; Engineering, Electrical & Electronic

WOS:000988411700001

IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC

20232114132110

Computer vision ; Image enhancement ; Integrated circuits ; Job analysis ; Timing circuits

Pulse Circuits:713.4 ; Semiconductor Devices and Integrated Circuits:714.2 ; Computer Applications:723.5 ; Vision:741.2

Web of Science

1.Univ Elect Sci & Technol China, Future Media Ctr, Chengdu 611731, Peoples R China
2.Univ Elect Sci & Technol China, Sch Comp Sci & Engn, Chengdu 611731, Peoples R China
3.Univ Elect Sci & Technol China, Sichuan Prov People Hosp, Chengdu 611731, Peoples R China
4.Southern Univ Sci & Technol, Dept Comp Sci & Engn, Shenzhen 518055, Peoples R China

Dai, Yan,Wang, Xuanhan,Gao, Lianli,et al. Overcoming Data Deficiency for Multi-Person Pose Estimation[J]. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS,2023,PP(99):1-12.

Dai, Yan,Wang, Xuanhan,Gao, Lianli,Song, Jingkuan,Zheng, Feng,&Shen, Heng Tao.(2023).Overcoming Data Deficiency for Multi-Person Pose Estimation.IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS,PP(99),1-12.

Dai, Yan,et al."Overcoming Data Deficiency for Multi-Person Pose Estimation".IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS PP.99(2023):1-12.