PathTR: Context-Aware Memory Transformer for Tumor Localization in Gigapixel Pathology Images

With the development of deep learning and computational pathology, whole-slide images (WSIs) are widely used in clinical diagnosis. A WSI, which refers to the scanning of conventional glass slides into digital slide images, usually contains gigabytes of pixels. Most existing methods in computer vision process WSIs as many individual patches, where the model infers the patches one by one to synthesize the final results, neglecting the intrinsic WSI-wise global correlations among the patches. In this paper, we propose the PATHology TRansformer (PathTR), which utilizes the global information of WSI combined with the local ones. In PathTR, the local context is first aggregated by a self-attention mechanism, and then we design a recursive mechanism to encode the global context as additional states to build the end to end model. Experiments on detecting lymph-node tumor metastases for breast cancer show that the proposed PathTR achieves the Free-response Receiver Operating Characteristic Curves (FROC) score of 87.68%, which outperforms the baseline and NCRF method with +8.99% and +7.08%, respectively. Our method also achieves a significant 94.25% sensitivity at 8 false positives per image.