Patcher: Patch Transformers with Mixture of
Experts for Precise Medical Image Segmentation
Yanglan Ou (1), Ye Yuan (2), Xiaolei Huang (1), Stephen T.C. Wong (3),
John Volpi (4), James Z. Wang (1), Kelvin Wong (3)
(1) The Pennsylvania State University, University Park, Pennsylvania, USA
(2) Carnegie Mellon University, Pittsburgh, Pennsylvania, USA
(3) TT and WF Chao Center for BRAIN & Houston Methodist Cancer Center,
Houston Methodist Hospital, Houston, Texas, USA
(4) Eddy Scurlock Comprehensive Stroke Center, Department of Neurology,
Houston Methodist Hospital, Houston, Texas, USA
Abstract:
We present a new encoder-decoder Vision Transformer architecture,
Patcher, for medical image segmentation. Unlike standard Vision
Transformers, it employs Patcher blocks that segment an image into large
patches, each of which is further divided into small patches. Transformers
are applied to the small patches within a large patch, which constrains
the receptive field of each pixel. We intentionally make the large patches
overlap to enhance intra-patch communication. The encoder employs a
cascade of Patcher blocks with increasing receptive fields to extract features
from local to global levels. This design allows Patcher to benefit
from both the coarse-to-fine feature extraction common in CNNs and the
superior spatial relationship modeling of Transformers. We also propose
a new mixture-of-experts (MoE) based decoder, which treats the feature
maps from the encoder as experts and selects a suitable set of expert features
to predict the label for each pixel. The use of MoE enables better
specializations of the expert features and reduces interference between
them during inference. Extensive experiments demonstrate that Patcher
outperforms state-of-the-art Transformer- and CNN-based approaches
significantly on stroke lesion segmentation and polyp segmentation. Code
for Patcher is released to facilitate related research.
Full Paper
(PDF, 10MB)
Source Codes
(GitHub)
More information
Citation:
Yanglan Ou, Ye Yuan, Xiaolei Huang, Stephen T.C. Wong, John Volpi,
James Z. Wang and Kelvin Wong, ``Patcher: Patch Transformers with
Mixture of Experts for Precise Medical Image Segmentation,''
Proceedings of the International Conference on Medical Image Computing
and Computer Assisted Interventions, Lecture Notes in Computer
Science, vol. 13435, Linwei Wang et al. (eds.), pp. 475-484,
Singapore, September 2022.
© 2022 MICCAI. Personal use of this material is permitted. However,
permission to reprint/republish this material for advertising or
promotional purposes or for creating new collective works for resale
or redistribution to servers or lists, or to reuse any copyrighted
component of this work in other works must be obtained from the MICCAI.
Last Modified:
September 14, 2022
© 2022