Accurate and Efficient Event-Based Semantic Segmentation Using Adaptive Spiking Encoder–Decoder Network

Rui Zhang1; Luziwei Leng2; Kaiwei Che3; Hu Zhang1; Jie Cheng2; Qinghai Guo2; Jianxing Liao2; Ran Cheng1

2024

10.1109/TNNLS.2024.3437415

IEEE Transactions on Neural Networks and Learning Systems   影响因子和分区

2162-2388

Spiking neural networks (SNNs), known for their low-power, event-driven computation, and intrinsic temporal dynamics, are emerging as promising solutions for processing dynamic, asynchronous signals from event-based sensors. Despite their potential, SNNs face challenges in training and architectural design, resulting in limited performance in challenging event-based dense prediction tasks compared with artificial neural networks (ANNs). In this work, we develop an efficient spiking encoder–decoder network (SpikingEDN) for large-scale event-based semantic segmentation (EbSS) tasks. To enhance the learning efficiency from dynamic event streams, we harness the adaptive threshold which improves network accuracy, sparsity, and robustness in streaming inference. Moreover, we develop a dual-path spiking spatially adaptive modulation (SSAM) module, which is specifically tailored to enhance the representation of sparse events and multimodal inputs, thereby considerably improving network performance. Our SpikingEDN attains a mean intersection over union (MIoU) of 72.57% on the DDD17 dataset and 58.32% on the larger DSEC-Semantic dataset, showing competitive results to the state-of-the-art ANNs while requiring substantially fewer computational resources. Our results shed light on the untapped potential of SNNs in event-based vision applications. The source codes are publicly available at https://github.com/EMI-Group/spikingedn.

第一

1.Department of Computer Science and Engineering, Shenzhen Key Laboratory of Computational Intelligence, Southern University of Science and Technology, Shenzhen, China
2.ACSLab, Huawei Technologies Company Ltd, Shenzhen, China
3.Department of Electrical and Electronic Engineering, Southern University of Science and Technology, Shenzhen, China

Rui Zhang,Luziwei Leng,Kaiwei Che,等. Accurate and Efficient Event-Based Semantic Segmentation Using Adaptive Spiking Encoder–Decoder Network[J]. IEEE Transactions on Neural Networks and Learning Systems,2024,PP(99).

Rui Zhang.,Luziwei Leng.,Kaiwei Che.,Hu Zhang.,Jie Cheng.,...&Ran Cheng.(2024).Accurate and Efficient Event-Based Semantic Segmentation Using Adaptive Spiking Encoder–Decoder Network.IEEE Transactions on Neural Networks and Learning Systems,PP(99).

Rui Zhang,et al."Accurate and Efficient Event-Based Semantic Segmentation Using Adaptive Spiking Encoder–Decoder Network".IEEE Transactions on Neural Networks and Learning Systems PP.99(2024).