DynamicKD: An effective knowledge distillation via dynamic entropy correction-based distillation for gap optimizing

Zhu，Songling1; Shang，Ronghua1; Yuan，Bo2; Zhang，Weitong1; Li，Wenjie1; Li，Yangyang1; Jiao，Licheng1

2024-09-01

10.1016/j.patcog.2024.110545

Pattern Recognition   影响因子和分区

0031-3203

The knowledge distillation uses a high-performance teacher network to guide the student network. However, the performance gap between the teacher and student networks can affect the student's training. This paper proposes a novel knowledge distillation algorithm based on dynamic entropy correction, which adjusts the student instead of the teacher to reduce the gap. Firstly, the effect of changing the output entropy (short for output information entropy) on the distillation loss in the student is analyzed in theory. This paper shows that correcting the output entropy can reduce the gap. Then, a knowledge distillation algorithm based on dynamic entropy correction is created, which can correct the output entropy in real-time with an entropy controller updated dynamically by the distillation loss. The proposed algorithm is validated on the CIFAR100, ImageNet, and PASCAL VOC 2007. The comparison with various state-of-the-art distillation algorithms shows impressive results, especially in the experiment on the CIFAR100 regarding teacher–student pair resnet32x4–resnet8x4. The proposed algorithm raises 2.64 points over the traditional distillation algorithm and 0.87 points over the state-of-the-art algorithm CRD in classification accuracy, demonstrating its effectiveness and efficiency.

CNN acceleration CNN compression Convolutional neural networks Knowledge distillation

SCI ; EI

英语

其他

20241916054022

Convolutional neural networks ; Personnel training ; Students

Chemical Operations:802.3 ; Personnel:912.4

ENGINEERING

2-s2.0-85192225231

Scopus

1.Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education,School of Artificial Intelligence,Xidian University,Xi'an, Shaanxi Province,710071,China
2.Guangdong Provincial Key Laboratory of Brain-inspired Intelligent Computation,Southern University of Science and Technology,Shenzhen,518055,China

Zhu，Songling,Shang，Ronghua,Yuan，Bo,et al. DynamicKD: An effective knowledge distillation via dynamic entropy correction-based distillation for gap optimizing[J]. Pattern Recognition,2024,153.

Zhu，Songling.,Shang，Ronghua.,Yuan，Bo.,Zhang，Weitong.,Li，Wenjie.,...&Jiao，Licheng.(2024).DynamicKD: An effective knowledge distillation via dynamic entropy correction-based distillation for gap optimizing.Pattern Recognition,153.

Zhu，Songling,et al."DynamicKD: An effective knowledge distillation via dynamic entropy correction-based distillation for gap optimizing".Pattern Recognition 153(2024).