Categorical Relation-Preserving Contrastive Knowledge Distillation for Medical Image Classification

Xing，Xiaohan1; Hou，Yuenan2; Li，Hang3; Yuan，Yixuan4; Li，Hongsheng1; Meng，Max Q.H.1,5

10.1007/978-3-030-87240-3_16

2021

0302-9743

1611-3349

LECT NOTES ARTIF INT

12905

163-173

The amount of medical images for training deep classification models is typically very scarce, making these deep models prone to overfit the training data. Studies showed that knowledge distillation (KD), especially the mean-teacher framework which is more robust to perturbations, can help mitigate the over-fitting effect. However, directly transferring KD from computer vision to medical image classification yields inferior performance as medical images suffer from higher intra-class variance and class imbalance. To address these issues, we propose a novel Categorical Relation-preserving Contrastive Knowledge Distillation (CRCKD) algorithm, which takes the commonly used mean-teacher model as the supervisor. Specifically, we propose a novel Class-guided Contrastive Distillation (CCD) module to pull closer positive image pairs from the same class in the teacher and student models, while pushing apart negative image pairs from different classes. With this regularization, the feature distribution of the student model shows higher intra-class similarity and inter-class variance. Besides, we propose a Categorical Relation Preserving (CRP) loss to distill the teacher’s relational knowledge in a robust and class-balanced manner. With the contribution of the CCD and CRP, our CRCKD algorithm can distill the relational knowledge more comprehensively. Extensive experiments on the HAM10000 and APTOS datasets demonstrate the superiority of the proposed CRCKD method. The source code is available at https://github.com/hathawayxxh/CRCKD.

Contrastive learning Knowledge distillation Medical image classification

其他

英语

CPCI-S ; EI

Computer Science ; Engineering ; Medical Informatics ; Imaging Science & Photographic Technology ; Radiology, Nuclear Medicine & Medical Imaging

Computer Science, Artificial Intelligence ; Computer Science, Software Engineering ; Engineering, Biomedical ; Medical Informatics ; Imaging Science & Photographic Technology ; Radiology, Nuclear Medicine & Medical Imaging

WOS:000712025900016

20214110994437

Computer vision ; Image classification ; Medical imaging

Biomedical Engineering:461.1 ; Data Processing and Image Processing:723.2 ; Computer Applications:723.5 ; Vision:741.2 ; Imaging Techniques:746 ; Chemical Operations:802.3

2-s2.0-85116469063

Scopus

1.Department of Electronic Engineering,The Chinese University of Hong Kong,Shatin,Hong Kong
2.Department of Information Engineering,The Chinese University of Hong Kong,Shatin,Hong Kong
3.School of Informatics,Xiamen University,Xiamen,China
4.Department of Electrical Engineering,City University of Hong Kong,Kowloon,Hong Kong
5.Department of Electronic and Electrical Engineering,Southern University of Science and Technology,Shenzhen,China

Xing，Xiaohan,Hou，Yuenan,Li，Hang,et al. Categorical Relation-Preserving Contrastive Knowledge Distillation for Medical Image Classification[C],2021:163-173.