Joint Classification of Hyperspectral Image and LiDAR Data Based on Spectral Prompt Tuning

Yi Kong1; Yuhu Cheng1; Yang Chen2; Xuesong Wang1

2024

10.1109/TGRS.2024.3417475

IEEE Transactions on Geoscience and Remote Sensing   影响因子和分区

1558-0644

The pretrained vision-language models (VLMs) have achieved outstanding performance in various visual tasks, primarily due to the knowledge they have acquired from massive image-text pairs. This enables VLMs to generalize to a wide range of downstream tasks. This article presents the first attempt to adapt VLMs for the joint classification task of hyperspectral image (HSI) and LiDAR data, aiming to leverage the well-learned VLMs to extract more generalizable features from diverse remote sensing image sources. Initially, using a patch encoder (PE), low-dimensional patches of HSI and LiDAR data are transformed into high-dimensional latent feature representations, meeting the dimensional requirements of VLMs for visual input data. Unlike traditional classifiers that rely on discrete class labels, VLM-based classification methods depend on continuous vectors, which can be derived from textual templates with class names, i.e., prompts. The classification performance of VLM-based methods heavily relies on these prompts, but prompt engineering not only demands extensive expert knowledge but also is extremely time-consuming. To address this, prompt tuning (PT) methods are introduced to enhance the generalizability of VLMs by adding spectral-based prompts to the vision encoder and incorporating randomly initialized, learnable text prompts (TPs) into the text encoder. Finally, through a novel class-discriminative loss function, the distance between text features of different classes is increased, thereby enhancing the model’s discriminative ability. Experimental results on the Houston 2013, Trento, and MUUFL datasets demonstrate that the proposed method can achieve competitive classification accuracy with a limited number of labeled pixels.

SCI ; EI

其他

GEOSCIENCES

1.Engineering Research Center of Intelligent Control for Underground Space, Ministry of Education, and the School of Information and Control Engineering, China University of Mining and Technology, Xuzhou, China
2.Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China

Yi Kong,Yuhu Cheng,Yang Chen,et al. Joint Classification of Hyperspectral Image and LiDAR Data Based on Spectral Prompt Tuning[J]. IEEE Transactions on Geoscience and Remote Sensing,2024,62.

Yi Kong,Yuhu Cheng,Yang Chen,&Xuesong Wang.(2024).Joint Classification of Hyperspectral Image and LiDAR Data Based on Spectral Prompt Tuning.IEEE Transactions on Geoscience and Remote Sensing,62.

Yi Kong,et al."Joint Classification of Hyperspectral Image and LiDAR Data Based on Spectral Prompt Tuning".IEEE Transactions on Geoscience and Remote Sensing 62(2024).