Fusing Metadata and Dermoscopy Images for Skin Disease Diagnosis

To date, it is still difficult and challenging to automatically classify dermoscopy images. Although the state-of-the-art convolutional networks were applied to solve the classification problem and achieved overall decent prediction results, there is still room for performance improvement, especially for rare disease categories. Considering that human dermatologists often make use of other information (e.g., body locations of skin lesions) to help diagnose, we propose using both dermoscopy images and non-image metadata for intelligent diagnosis of skin diseases. Specifically, the metadata information is innovatively applied to control the importance of different types of visual information during diagnosis. Comprehensive experiments with various deep learning model architectures demonstrated the superior performance of the proposed fusion approach especially for relatively rare diseases. All our codes will be made publicly available(1).