Transformers for Medical Image Segmentation

Transformers have emerged as a powerful class of deep learning models that overcome the locality constraints of convolutional neural networks by modelling long-range dependencies through self-attention mechanisms. In medical image segmentation, these architectures capture global contextual information vital for delineating complex anatomical structures and pathological regions across diverse modalities such as magnetic resonance imaging, computed tomography and ultrasound. Hybrid designs integrate convolutional encoders for efficient feature extraction with transformer blocks to establish pixel-wise relationships at scale, resulting in improved boundary precision and robustness to variations in lesion size, shape and imaging artefacts. Vision Transformer adaptations and windowed self-attention schemes preserve high-resolution spatial details, while multi-scale feature exchange facilitates segmentation of both large organs and fine structures. Such advances hold promise for clinical workflows including tumour volume quantification, surgical guidance and multi-organ mapping, paving the way for diagnostic consistency, reproducibility and real-time decision support in radiology and pathology.

Research from Nature Portfolio

No recent Nature Portfolio content available.

Research from all publishers

A high-resolution Swin Transformer network replaces convolutional stages with shifted-window transformer blocks arranged in a high-resolution pathway, continuously exchanging features at multiple scales. This design preserves spatial precision and achieves segmentation performance on par with state-of-the-art U-Net-like hybrids for brain tumour, liver and multi-organ tasks while maintaining high fidelity in boundary regions.

A family of models termed Factorizer integrates nonnegative matrix factorisation within a U-shaped transformer architecture to approximate global context in a linearly scalable manner. By combining low-rank context modelling with shifted-window attention, these networks deliver competitive accuracy on brain tumour and stroke lesion benchmarks, offering enhanced interpretability through meaningful factor components and accelerated inference without sacrificing segmentation quality.

A three-dimensional bi-directional transformer U-Net (3DTU) embeds a 3D transformer in both encoder and decoder pathways to capture volumetric dependencies for MRI and CT segmentation. This end-to-end framework demonstrates superior performance across multiple 3D datasets by jointly leveraging local convolutional reasoning and volumetric self-attention, improving delineation of intricate anatomical regions in three-dimensional scans.

Transformers for Medical Image Segmentation publication trend

The graph below shows the total number of articles in transformers for medical image segmentation across all publications each year (not limited to Nature Index journals).

Technical terms

Transformer: A neural network architecture that employs self-attention layers to model global relationships between all elements in an input sequence, enabling long-range dependency learning.

Self-attention: A mechanism that computes pairwise interactions between positions of an input, allowing the model to weigh and integrate information from distant regions within an image.

U-shaped architecture (U-Net): A segmentation framework consisting of an encoder that extracts features and a decoder that reconstructs spatial details, with skip connections linking corresponding levels to preserve localisation.

Dice Similarity Coefficient (DSC): A statistical metric measuring overlap between predicted and ground truth segmentation masks, ranging from 0 (no overlap) to 1 (perfect agreement).

References

1. High-Resolution Swin Transformer for Automatic Medical Image Segmentation. Sensors (2023).
2. Factorizer: A scalable interpretable approach to context modeling for medical image segmentation. Medical Image Analysis (2022).
3. 3D bi-directional transformer U-Net for medical image segmentation. Frontiers in Big Data (2023).