Multi-Scale Features Are Effective for Multi-Modal Classification: An Architecture Search Viewpoint

Pinhan Fu1; Xinyan Liang1; Yuhua Qian1; Qian Guo2; Yayu Zhang1; Qin Huang1; Ke Tang3

2024

10.1109/TCSVT.2024.3470996

IEEE Transactions on Circuits and Systems for Video Technology   影响因子和分区

1558-2205

Multi-modal neural architecture search (MNAS) is an effective approach to obtain task-adaptive multi-modal classification models. Deep neural networks, as currently mainstream feature extractors, can provide hierarchical features for each modality. Existing MNAS methods face difficulty in exploiting such hierarchical features due to their different form coexistence such as tensorial multi-scale features and vectorized penultimate features. Moreover, existing methods always focus on the evolution of fusion operators or vectorized features of all modalities, constraining search space. In this paper, a novel two-stage method called multi-modal multi-scale evolutionary neural architecture search (MM-ENAS) is proposed. The first stage unifies the representation form of hierarchical features by the proposed evolutionary statistics strategy. The second stage identifies the optimal combination of basic fusion operations for all unified hierarchical features by the evolutionary algorithm. MM-ENAS increases search space by simultaneously searching for feature statistical extraction methods, basic fusion operators and feature representation set consisting of tensorial multi-scale features and vectorized penultimate features. Experimental results on three multi-modal tasks demonstrate that the proposed method achieves competitive performance in terms of accuracy, search time, and number of parameters compared to existing representative MNAS methods. Additionally, the method exhibits fast adaptation to various multi-modal tasks.

其他

1.Institute of Big Data Science and Industry, Shanxi University, Taiyuan, Shanxi, China
2.School of Computer Science and Technology, Taiyuan University of Science and Technology, Taiyuan, Shanxi, China
3.the Department of Computer Science and Engineering, Shenzhen Key Laboratory of Computational Intelligence, Southern University of Science and Technology, Shenzhen, Guangdong, China

Pinhan Fu,Xinyan Liang,Yuhua Qian,et al. Multi-Scale Features Are Effective for Multi-Modal Classification: An Architecture Search Viewpoint[J]. IEEE Transactions on Circuits and Systems for Video Technology,2024,PP(99).

Pinhan Fu.,Xinyan Liang.,Yuhua Qian.,Qian Guo.,Yayu Zhang.,...&Ke Tang.(2024).Multi-Scale Features Are Effective for Multi-Modal Classification: An Architecture Search Viewpoint.IEEE Transactions on Circuits and Systems for Video Technology,PP(99).

Pinhan Fu,et al."Multi-Scale Features Are Effective for Multi-Modal Classification: An Architecture Search Viewpoint".IEEE Transactions on Circuits and Systems for Video Technology PP.99(2024).