Introduction

Magnetic Resonance Imaging (MRI) is a widely used diagnostic and research tool in clinical settings, offering high-resolution imaging and diverse contrast mechanisms to visualize various structural and functional characteristics of the underlying anatomy. However, one of the significant limitations of MRI is its relatively long acquisition time, which can reduce patient throughput, increase costs, and lead to delays in diagnosis. Long scan times can also contribute to patient discomfort and motion-related artifacts, which may degrade image quality. To address this challenge, various acceleration techniques have been developed, with compressed sensing (CS)1,2,3 being one of the most widely adopted methods. CS accelerates MRI acquisition by undersampling k-space data, relying on sparsity constraints and iterative algorithms to recover high-quality images from limited measurements. While effective, CS-based approaches often introduce reconstruction errors, including aliasing artifacts and loss of fine structural details, particularly in high-acceleration settings.

Recent advances in deep learning have provided powerful alternatives for accelerated MRI reconstruction4,5,6,7,8,9,10. Deep learning-based methods leverage large datasets to learn complex mappings between undersampled and fully sampled k-space or image domain representations. Many state-of-the-art deep learning models for accelerated MRI are based on convolutional neural networks (CNNs). These CNN-based architectures have demonstrated remarkable improvements over traditional CS-based methods, yielding higher-quality with fewer artifacts and faster inference times. However, CNNs have inherent limitations due to their local receptive fields and translation-invariant convolutional operations11,12. These characteristics can restrict their ability to capture long-range dependencies and global contextual information, leading to suboptimal performance, especially in cases where high-frequency details are critical13.

The emergence of transformer architectures has introduced new possibilities for image processing and computer vision tasks. Originally developed for natural language processing (NLP), transformers14 and their variants have been widely applied to tasks such as text classification, machine translation, and question-answering15,16,17,18,19,20. A key advantage of transformer-based models is their ability to capture long-range dependencies and contextual information via self-attention mechanisms21,22. Inspired by this, Vision Transformer (ViT)23 adapted transformers for image-related tasks by treating input images as sequences of non-overlapping patches, similar to words in NLP. Unlike CNNs, which gradually expands the receptive field hierarchically, even a shallow ViT model can effectively model global contextual relationships, making it highly competitive across various vision applications24,25,26.

Further developments in transformer-based models have led to innovations such as masked autoencoders (MAE)27, which utilizes a masked token prediction strategy and encoder-decoder structure during pretraining to enhance representation learning. These techniques have demonstrated strong potential in image reconstruction tasks, where learning robust feature representations is crucial. Additionally, models like the Segment Anything Model (SAM)28,29 have demonstrated the adaptability of ViT-based structures as backbones for multimodal learning, further expanding their applicability to medical imaging.

Despite these advancements, applying transformer-based models to accelerated MRI remains an area of active research. Several studies have explored ViT-based architectures for accelerated MRI, showing that transformer models can benefit from pretraining on large datasets and outperform CNN-based approaches in certain settings30,31. However, most of these efforts focus on task-specific models rather than developing a generalized framework32. Existing ViT-based models are often designed for specific undersampling patterns and acceleration factors, limiting their adaptability across different acquisition setups. A more generalizable approach is needed to fully leverage the capabilities of transformers for accelerated MRI.

Image Processing Transformer (IPT)33 has emerged as a promising framework for achieving generalizability in low-level imaging tasks. IPT introduces a multi-task learning paradigm by incorporating multiple input-output configurations within a single framework. By integrating multiple heads and tails for different image processing tasks, the shared transformer body learns to extract universal feature representations, improving model adaptability across diverse imaging scenarios. This design has demonstrated success in generalizing across multiple low-level image processing tasks such as denoising, deraining, and super-resolution.

Motivated by these developments, we introduce the Magnetic Resonance Image Processing Transformer (MR-IPT), a novel image-domain framework designed to enhance the generalizability of ViT-based models for accelerated MRI restoration. MR-IPT extends the IPT paradigm by interpreting different undersampling restoration setups as distinct tasks, allowing the core ViT backbone to focus on learning robust feature representations. We pre-train MR-IPT on a large-scale medical imaging dataset to maximize its feature extraction capabilities. Subsequently, we evaluate its performance on multiple downstream MRI restoration tasks, incorporating various acceleration factors and sampling masks to assess its adaptability. Our experimental results demonstrate that MR-IPT outperforms both CNN- and ViT-based models across a range of MRI restoration scenarios. Notably, MR-IPT exhibits strong generalization capabilities, effectively handling unseen sampling patterns and acceleration rates. Additionally, we conduct model stability assessments, showing that MR-IPT maintains high quality even when trained with limited downstream data. These findings highlight the potential of MR-IPT as a robust and scalable solution for accelerated MRI restoration.

Results

MR-IPT framework

The MR-IPT framework consists of five core components: heads, tails, a prompt encoder, a shared encoder, and a shared decoder (Fig. 1a). The shared encoder utilizes shifted-window multi-head self-attention (W-MSA)34 to efficiently capture global context across multiple layers. Inspired by MAE and SAM, we implemented a lightweight decoder incorporating prompt self-attention and two-way cross-attention, facilitating effective feature refinement and restoration. This lightweight design allows for a deeper encoder architecture without significantly increasing model size and computational costs, thereby enhancing the model’s representational capacity. The heads extract features from undersampled images, transforming them into patch tokens. The prompt encoder generates prompt tokens based on acceleration labels, which, together with image patch tokens, are processed by the shared encoder-decoder to recover missing information. The tails then reconstruct fully sampled images from the learned restoration tokens (Fig. 1b). To ensure broad generalizability in accelerated MRI restoration, we train MR-IPT on images corrupted using a diverse range of sampling masks and acceleration ratios, as illustrated in Fig. 1c.

Unlike the original IPT, which assigns a dedicated head-tail pair to each specific restoration task, we implement three MR-IPT variants across diverse undersampling patterns: (1) MR-IPT-type: Heads and tails are aggregated based on acceleration ratios, where each head-tail pair specializes in reconstructing images from different sampling masks; (2) MR-IPT-level: Heads and tails are aggregated based on sampling masks, allowing each head-tail pair to focus on restoration across different acceleration ratios; (3) MR-IPT-split: Each unique combination of sampling mask and acceleration ratio is assigned a dedicated head-tail pair.

To fully leverage the generalization potential of MR-IPT, we trained our model on RadImageNet35, a large-scale medical imaging dataset. Training images were corrupted at five levels, with sampling ratios ranging from two to ten, incorporating both 1D and 2D sampling masks to enhance robustness. For evaluation, we conducted downstream MRI restoration experiments on the fastMRI dataset36, assessing MR-IPT’s performance across typical restoration scenarios, unseen sampling ratios and masks, zero-shot generalization capabilities, and model stability under limited data conditions.

Fig. 1
figure 1

The overall architecture of the Magnetic Resonance Image Processing Transformer (MR-IPT) framework. (a) MR-IPT consists of heads, tails, a prompt encoder, a shared encoder, and a shared decoder. (b) A lightweight decoder incorporating prompt self-attention and two-way cross-attention enhances feature refinement and restoration. This design enables a deeper encoder architecture without significantly increasing model size or computational cost, improving the model’s representational capacity. The heads extract features from undersampled images and transform them into patch tokens. The prompt encoder generates prompt tokens based on acceleration labels, which, together with image patch tokens, are processed by the shared encoder-decoder to recover missing information. The tails reconstruct fully sampled images from the learned restoration tokens. (c) During pre-training, MR-IPT is trained on diverse acceleration ratios (2×, 4×, 6×, 8×, and 10×) and various sampling masks. The 1D sampling masks include Cartesian random, Cartesian equispaced, and 1D Gaussian, while the 2D sampling mask follows a 2D Gaussian distribution.

Full size image

Accelerated MRI restoration performance

In this section, we evaluate MR-IPT on accelerated MRI restoration tasks using the fastMRI knee and brain datasets. Following the standard fastMRI benchmark, knee dataset restorations are performed with 4× and 8× Cartesian random undersampling, while brain dataset restorations are tested at 4× and 8× Cartesian equispaced undersampling.

We compare MR-IPT against multiple representative models, including UNet3237, UNet128, ViT-Base, ViT-L23,30, E2E-VarNet9, and ReconFormer31. Additionally, to assess the impact of large-scale pretraining, we introduce UNet128-FT and ViT-L-FT, both of which are pretrained on RadImageNet (the same as MR-IPT) before being fine-tuned on the corresponding fastMRI datasets. Note that models like E2E-VarNet are optimized for parallel imaging, the application with single-coil data could limit the network’s capacity to exploit multi-coil sensitivity maps. The quantitative results are presented in Table 1, where MR-IPT consistently outperforms baseline models across different acceleration ratios. Among its three variants, MR-IPT-level exhibits slightly superior performance compared to MR-IPT-type and MR-IPT-split, suggesting that aggregating heads and tails by sampling mask provides an effective balance between task-specific optimization and generalizability.

For instance, in the 4× brain restoration task, MR-IPT-level achieves a PSNR/SSIM of 42.48/0.9831, outperforming UNet128 (36.25/0.9648), ViT-L (37.54/0.9558) and ReconFormer (41.63/0.9737). When compared to pretrained and fine-tuned models (UNet128-FT: 37.27/0.9653, ViT-L-FT: 37.54/0.9558), MR-IPT still demonstrates a clear advantage. This highlights the effectiveness of MR-IPT’s multi-head-tail structure and its unified shared encoder-decoder, which maximizes the benefits of large-scale pretraining by improving feature adaptability across different undersampling conditions.

Figure 2 illustrates qualitative comparisons of reconstructed images, including error maps that represent absolute differences between restorations and ground truth images (intensified by a factor of three for better visualization). Figure 3 provides a comparative analysis of the three MR-IPT variants (MR-IPT-type, MR-IPT-level, and MR-IPT-split) under 4× and 8× Cartesian random and Cartesian equispaced undersampling. Overall, MR-IPT produces cleaner error maps across all tested sampling ratios and masks. However, a closer look reveals that some methods, such as ReconFormer, can produce sharper details especially with higher acceleration factor due to their specialized modules for enforcing data-consistency (e.g., the 8× Cartesian random knee image in Fig. 2). Conversely, the MR-IPT restorations can sometimes appear smoother. When evaluating images with pathological features, such as the 8× Cartesian equispaced brain image in Fig. 2, all tested networks struggle to accurately recover finer structures, like the tumor, and show suboptimal performance compared to the fully sampled image. This highlights a critical area for future research focusing on pathology-specific evaluation.

Table 1 Restoration performance comparison on the FastMRI knee and brain datasets. We report peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) on the test sets, evaluating the quality of reconstructed images against fully sampled ground truth images.
Full size table
Fig. 2
figure 2

Restoration comparison across different models. Each column presents reconstructed images from various methods, highlighting differences in image quality and artifact suppression. The second column of each subplot shows the corresponding error maps (intensified by a factor of three for better visualization), which visualize absolute differences between the reconstructed images and the fully sampled ground truth. The red boxes highlight areas of differences among models.

Full size image
Fig. 3
figure 3

Comparison of MR-IPT variants across different undersampling patterns. We implement three MR-IPT variants across diverse undersampling patterns: (1) MR-IPT-type, where heads and tails are grouped based on acceleration ratios, with each head-tail pair specializing in different sampling masks; (2) MR-IPT-level, where heads and tails are grouped based on sampling masks, allowing each pair to generalize across different acceleration ratios; and (3) MR-IPT-split, where each unique combination of sampling mask and acceleration ratio is assigned a dedicated head-tail pair. The results demonstrate that all three variants achieve high-quality restorations, highlighting MR-IPT’s flexibility in handling diverse undersampling patterns.

Full size image

Performance on new sampling ratios

Given that downstream tasks may involve different undersampling ratios, it is crucial to assess MR-IPT’s generalization to previously unseen acceleration factors. To keep a balance between the number of head-tail pairs and overall generalizability, we pre-train MR-IPT using five acceleration ratios (2×, 4×, 6×, 8×, and 10×), covering a broad range of undersampling scenarios. To further evaluate its adaptability, we conduct downstream restoration on the brain dataset with unseen acceleration ratios of 5× and 7× during inference.

Table 2 presents the quantitative results. UNet128 and ViT-L are trained directly on the indicated acceleration ratios, whereas UNet128-FT and ViT-L-FT follow the same pretraining-finetuning pipeline as MR-IPT for a fair comparison. Notably, both UNet128 and ViT-L benefit from pretraining, with ViT-L-FT showing greater improvements, achieving PSNR/SSIM of 36.29/0.9490 (5×) and 34.60/0.9378 (7×), compared to UNet128-FT at 35.52/0.9554 (5×) and 32.96/0.9346 (7×). MR-IPT-level consistently outperforms both models, achieving PSNR/SSIM of 39.92/0.9763 (5×) and 36.69/0.9626 (7×), highlighting its superior restoration capability.

Figure 4 provides qualitative comparisons of reconstructed images. Interestingly, while ViT-L-FT and UNet128-FT exhibit differences in error map characteristics, UNet128-FT, despite having a higher maximum absolute error, produces cleaner backgrounds, particularly at higher acceleration ratios where overall image intensity is lower. MR-IPT consistently yields the cleanest error maps across all tested setups, including unseen sampling ratios, demonstrating its strong adaptability and generalization to novel acceleration factors.

Table 2 Restoration performance comparison on unseen sampling ratios (5× and 7×) for the brain dataset. UNet128-FT and ViT-L-FT follow the same pretraining-finetuning pipeline as MR-IPT to ensure a fair evaluation of generalization performance. Results demonstrate MR-IPT’s superior adaptability to novel acceleration factors, outperforming both UNet128-FT and ViT-L-FT in PSNR and SSIM.
Full size table
Fig. 4
figure 4

Restoration comparison across multiple sampling ratios, including unseen configurations. This figure showcases the performance of different models in reconstructing MRI images at various acceleration ratios. Results highlighted in the yellow block represent new sampling ratios (e.g., 5× and 7×) that were not encountered during pre-training, demonstrating each model’s generalization ability. The results highlight MR-IPT’s strong adaptability to previously unseen acceleration factors, effectively preserving fine anatomical structures while minimizing artifacts, compared to other baseline methods.

Full size image

Performance on new sampling masks

In this section, we evaluate MR-IPT’s adaptability to novel sampling masks. As depicted in Fig. 1c, our pretraining strategy includes a diverse set of 1D (Cartesian random, Cartesian equispaced, and 1D Gaussian) and 2D (2D Gaussian) sampling masks. To assess the impact of excluding 2D masks during pretraining, we introduced three MR-IPT-1D variants, trained exclusively on 1D masks. For downstream evaluations, we test all models on accelerated MRI resotrations using 4× and 8 × 2D Gaussian sampling masks.

Table 3 presents quantitative results comparing standard MR-IPT, MR-IPT-1D, and baseline models UNet128-FT and ViT-L-FT. Notably, both UNet128-FT and ViT-L-FT are pretrained with 2D Gaussian masks, aligning with the standard MR-IPT setup. Surprisingly, despite being trained solely on 1D masks, MR-IPT-1D demonstrates strong generalization and competitive performance on 2D-masked restorations. For instance, MR-IPT-level-1D achieves a PSNR/SSIM of 37.99/0.9685 (8×), closely matching MR-IPT-level at 38.94/0.9727 and significantly surpassing UNet128-FT (32.30/0.9310) and ViT-L-FT (33.21/0.9373). Interestingly, when evaluating 1D-masked restorations, MR-IPT-1D slightly outperforms the standard MR-IPT due to its specialized pretraining. For example, in 8× Cartesian equispaced restoration, MR-IPT-level-1D achieves a PSNR/SSIM of 35.60/0.9562, compared to MR-IPT-level at 35.53/0.9557. Figure 5 provides qualitative comparisons of reconstructed images, demonstrating that both MR-IPT and MR-IPT-1D generate cleaner error maps for 2D-masked inputs than ViT-L-FT, further validating the robustness and adaptability of the MR-IPT framework.

Table 3 Restoration performance comparison on unseen sampling masks. MR-IPT-1D models are pretrained exclusively on 1D masks and evaluated on both 1D and 2D-masked restorations to assess cross-mask generalization. Standard MR-IPT, UNet128-FT, and ViT-L-FT serve as baselines. Results highlight MR-IPT-1D’s ability to adapt to novel 2D masks while maintaining strong performance on 1D-masked restorations.
Full size table
Fig. 5
figure 5

Restoration comparison using 2D Gaussian sampling masks. Results highlighted in the green block correspond to MR-IPT-1D models, which were pre-trained exclusively on 1D sampling masks. Despite the absence of 2D masks during pre-training, MR-IPT-1D demonstrates strong generalization capabilities, achieving high-fidelity restorations with minimal artifacts, further validating its robustness for new sampling masks.

Full size image

Zero-shot performance

In this part, we evaluate the zero-shot performance of MR-IPT in comparison to other models. The quantitative results are summarized in Table 4. All models are pretrained on the same dataset before being tested directly on the specified acceleration setups without additional fine-tuning. Since UNet and ViT do not include a multi-head-tail structure for different sampling masks and ratios, during pretraining, sampling masks and ratios were randomly selected for each image in a batch. Across all tested configurations, MR-IPT outperforms both UNet128 and ViT-L by a significant margin, demonstrating superior generalization ability. For example, in the 8× Cartesian random sampling test, MR-IPT-level achieves a PSNR/SSIM of 30.65/0.7814, surpassing UNet128 at 27.48/0.7184 and ViT-L at 29.00/0.7296. Similarly, in the 4× Cartesian equispaced test, MR-IPT-level attains a PSNR/SSIM of 39.90/0.9756, outperforming UNet128 at 32.89/0.9315 and ViT-L at 35.73/0.9489. These results highlight the effectiveness of MR-IPT’s multi-head-tail design and shared encoder-decoder architecture in adapting to novel sampling patterns without task-specific fine-tuning.

Table 4 Zero-shot restoration performance comparison. All models are pretrained on the same dataset and evaluated directly on the specified acceleration setups without additional fine-tuning.
Full size table

Model stability regarding downstream dataset size

In many real-world scenarios, acquiring large-scale medical imaging datasets for comprehensive training of CNN-based models is challenging. When only a limited number of images are available, fine-tuning MR-IPT on a small dataset becomes a more practical approach. To evaluate MR-IPT’s performance under varying dataset sizes, we fine-tuned our model on the fastMRI brain dataset using an 8× Cartesian equispaced mask, with dataset sizes ranging from 10 to 2500. For each size, we randomly sampled subsets from the dataset and repeated the process ten times to assess model performance and stability.

As illustrated in Fig. 6, both the average performance and stability of MR-IPT improve as dataset size increases. For instance, with just 10 training samples, MR-IPT achieves a PSNR/SSIM of 32.78/0.9316. As the dataset size increases to 2500, the performance improves to 34.05/0.9441, approaching the performance of MR-IPT fine-tuned on the full dataset (35.53/0.9557). These results highlight MR-IPT’s ability to leverage large-scale pretraining, maintaining stable and competitive restoration quality even in data-constrained settings.

Fig. 6
figure 6

MR-IPT performance across different dataset sizes. To assess the impact of dataset size on restoration quality, MR-IPT was fine-tuned on the fastMRI brain dataset using an 8× Cartesian equispaced mask, with training subsets ranging from 10 to 2500 images. Each subset was randomly sampled, and the process was repeated ten times to evaluate both model performance and stability. The red line and green line represent zero-shot and fully fine-tuned performance. The results indicate that as the dataset size increases, MR-IPT exhibits significant improvements in both accuracy and consistency. Even with a limited number of training samples, the model maintains competitive performance, demonstrating its ability to leverage large-scale pretraining effectively and achieve stable, high-quality restorations in data-constrained scenarios.

Full size image

Performance compared to IPT

We conducted a comparison between our MR-IPT model and the original IPT framework. Quantitative results are summarized in Table 5, with visual comparisons shown in Fig. 7. In IPT, the head-tail setup functions similarly to MR-IPT’s split mode, where each unique combination of sampling mask and acceleration ratio corresponds to a dedicated head-tail pair. We evaluate both zero-shot and fine-tuned performance for 4× and 8× undersampling with Cartesian random and Cartesian equispaced masks. Pre-trained on the same dataset, which includes both 1D and 2D sampling masks, MR-IPT generally outperforms IPT across various configurations. For example, in the 4× Cartesian random restoration on the knee dataset, MR-IPT achieves a PSNR/SSIM of 34.22/0.8635 (zero-shot) and 34.52/0.8681 (fine-tuned), whereas IPT performs at 31.61/0.8038 (zero-shot) and 32.70/0.8094 (fine-tuned). Similarly, for the 4× Cartesian equispaced restoration on the brain dataset, MR-IPT reaches a PSNR/SSIM of 39.90/0.9756 (zero-shot) and 42.48/0.9831 (fine-tuned), compared to IPT’s 37.63/0.9580 (zero-shot) and 39.43/0.9640 (fine-tuned).

Table 5 Comparison of MR-IPT and IPT performance. MR-IPT-level is used as the baseline for evaluation, with both zero-shot and fine-tuned results for various undersampling scenarios across different sampling masks.
Full size table
Fig. 7
figure 7

Comparison of MR-IPT and IPT restorations across different undersampling settings. This figure evaluates the performance of MR-IPT against the original Image Processing Transformer (IPT) under both fine-tuned and zero-shot scenarios. Results in the green block represent fine-tuned comparisons, where both IPT and MR-IPT were trained on 4× and 8× undersampling with Cartesian random and Cartesian equispaced masks. Results in the yellow block illustrate zero-shot comparisons, where models were tested without additional fine-tuning. The results demonstrate that MR-IPT consistently outperforms IPT, achieving higher restoration fidelity, better structural preservation, and reduced artifacts, highlighting its superior adaptability and generalization capabilities.

Full size image

Ablation studies and other analysis

To evaluate the effectiveness of the prompt encoder, we conducted ablation studies to analyze the performance on the fastMRI knee dataset with 4× and 8× Cartesian random sampling masks. As demonstrated in Table 6, excluding prompt encoding in MR-IPT leads to a performance drop of PSNR/SSIM from 34.52/0.8681 to 33.51/0.8283 in 4× restoration and 31.45/0.7952 to 30.96/0.7670 in 8× restoration.

In Fig. 8, we conducted further analysis regarding model performance of low (1/3 central region in k-space) and high (2/3 peripheral region in k-space) spatial frequencies on the fastMRI brain dataset with 4× Cartesian equispaced sampling masks. As demonstrated, MR-IPT achieves better performance in both cases with more accurate general shape and sharper details.

In Table 7, we demonstrate computational cost analysis with comparison of model throughputs. We performed our test on an RTX 4090 GPU during evaluation. Figure 9 illustrates model performance relative to model size for multiple downstream tasks, with the size of each model represented by the size of the corresponding marker. Overall, this test highlights that MR-IPT consistently outperforms IPT, ViT-L-FT, and UNet128-FT with similar model sizes. This demonstrates the effectiveness of MR-IPT’s lightweight decoder and prompt encoder design, which allows for a larger encoder and thus more effective latent space learning. In summary, our results show that MR-IPT provides robust and stable performance across a wide range of downstream tasks. Notably, it excels in handling new sampling ratios and masks and demonstrates impressive zero-shot generalization as well as stability when trained on limited dataset sizes.

Table 6 Impact analysis of the prompt encoding on the FastMRI knee dataset.
Full size table
Table 7 Comparison of model throughput. The test is conducted with an RTX 4090.
Full size table
Fig. 8
figure 8

(a) Frequency analysis comparison on the fastMRI brain dataset. (b) Restoration samples for MR-IPT in low frequency (1/3 central region in k-space) and high frequency (2/3 peripheral region in k-space). All high frequency images and error maps have been intensified three times for better demonstration.

Full size image
Fig. 9
figure 9

Model performance versus size across different restoration tasks. Each marker represents a model, with its size proportional to the model’s parameter count, illustrating the trade-off between computational complexity and restoration performance. The results show that MR-IPT consistently outperforms IPT, ViT-L-FT, and others, even when operating at comparable model sizes. This superior performance is attributed to MR-IPT’s lightweight decoder and prompt encoder design, which enables the use of a deeper and more expressive encoder for improved latent space learning without significantly increasing computational cost. The findings highlight MR-IPT’s efficiency, scalability, and effectiveness in MRI restoration across diverse undersampling conditions.

Full size image

Discussion

In this work, we introduce MR-IPT, a transformer-based framework designed for general accelerated MRI restoration. Unlike previous ViT-based models that primarily focus on task-specific restoration, MR-IPT is built to fully leverage the potential of ViT through large-scale pre-training. This approach enables MR-IPT to deliver superior performance across diverse accelerated MRI restoration tasks. Its multi-head-tail design provides flexibility, supporting a wide range of undersampling masks and acceleration ratios during image degradation. The shared ViT backbone facilitates a unified structure for latent space representation and learning under various sampling conditions. Additionally, the lightweight decoder, deeper encoder, and label prompt encoder contribute to enhanced restoration performance compared to conventional ViT models.

MR-IPT consistently demonstrates strong performance compared to traditional MRI restoration networks such as UNet and ViT. As illustrated in Table 1; Fig. 2, MR-IPT outperforms these models significantly. For example, in the 4-fold Cartesian equispaced sampling test on the fastMRI brain dataset, MR-IPT-level achieves a PSNR/SSIM of 42.48/0.9831, compared to UNet128 at 36.25/0.9648 and ViT-L at 37.54/0.9558. Even when employing the same pre-training strategies, MR-IPT maintains its advantage, outperforming both UNet128-FT (37.27/0.9653) and ViT-L-FT (37.63/0.9564).

MR-IPT also exhibits stable and superior performance when evaluated on new sampling masks and acceleration ratios, as shown in Tables 2 and 3; Figs. 4 and 5. In the Cartesian equispaced restoration test with a new 5× sampling ratio, MR-IPT-level achieves a PSNR/SSIM of 39.92/0.9763, outperforming UNet128-FT (35.52/0.9554) and ViT-L-FT (36.29/0.9490). Remarkably, in the 4 × 2D Gaussian sampling restoration test, where the model was pre-trained solely with 1D sampling masks, MR-IPT-level-1D achieves a PSNR/SSIM of 42.30/0.9839. This performance is comparable to MR-IPT pre-trained with both 1D and 2D masks (MR-IPT-level at 42.69/0.9849) and superior to UNet128-FT (35.62/0.9571) and ViT-L-FT (37.47/0.9638), both of which included 2D sampling masks during pre-training. These results underscore MR-IPT’s adaptability and generalizability in accelerated MRI restoration tasks.

Furthermore, MR-IPT demonstrates robust performance in zero-shot learning scenarios, as summarized in Table 4. For instance, in the 8× Cartesian equispaced test, MR-IPT-level achieves a PSNR/SSIM of 32.52/0.9288, outperforming UNet128 (27.93/0.8584) and ViT-L (30.21/0.9197). In situations with limited training data, MR-IPT maintains stable and high-quality restoration performance, as shown in Fig. 6. These results highlight MR-IPT’s ability to leverage large-scale pre-training effectively, ensuring competitive restoration quality even in data-constrained settings.

When compared to the original IPT framework, MR-IPT consistently exhibits superior performance in both zero-shot and fine-tuned scenarios, as demonstrated in Table 5; Fig. 7. For example, in the 4× Cartesian equispaced restoration of the brain dataset, MR-IPT achieves a PSNR/SSIM of 39.90/0.9756 (zero-shot) and 42.48/0.9831 (fine-tuned), compared to IPT’s 37.63/0.9580 (zero-shot) and 39.43/0.9640 (fine-tuned). Comparative studies (Fig. 9) further reveal that MR-IPT outperforms IPT, ViT-L-FT, and UNet128-FT even with similar model sizes. This superior performance is attributed to MR-IPT’s lightweight decoder and prompt encoder design, which enables a larger encoder for more effective latent space learning.

Despite its strong performance, this study has several limitations. First, our current framework operates as an image-domain restoration network and, as such, does not directly process raw k-space data. A comprehensive evaluation of its efficacy for multi-coil MRI would require extending the approach to work directly with clinical k-space data. This further raises another important topic regarding data discrepancy handling to leverage large-scale pre-training and downstream fine-tuning. For example, since a resource like RadImageNet is composed solely of images, it’s not immediately clear how a network pre-trained on it can be adapted for downstream multi-coil data. One potential avenue for future work is to explore how a pre-trained image-domain model could be integrated into a dual-domain pipeline. Alternatively, a separate adapting layer could be introduced during the fine-tuning process to bridge the gap between single-coil and multi-coil images. As an image-domain method, MR-IPT does not enforce explicit k-space data consistency. While this approach successfully leverages ViT-based multi-task pre-training to effectively suppress common artifacts and enhance image quality, the absence of a data fidelity constraint presents a limitation. Specifically, it can potentially introduce hallucination artifacts and may compromise quantitative fidelity, factors that are critical for clinical adoption. Consequently, the integration of data consistency mechanisms represents a vital direction for future work to improve the reliability and clinical applicability of the framework. Second, the dataset used for pre-training is limited to accelerated MRI data. Although previous studies have shown that pre-training on general image datasets can further enhance performance on accelerated MRI30, our pre-training dataset remains confined to medical imaging. Future work should investigate the benefits of incorporating more diverse datasets and explore the potential of merging multiple datasets to scale up pre-training, potentially improving model performance. In addition, we mainly use the fastMRI dataset for downstream restoration task evaluations, future studies could include other datasets for a more comprehensive generalizability investigation. Furthermore, our model’s performance is constrained by the fixed input size required for pre-training, which may not match the native resolution of all downstream clinical datasets. To improve generalizability, a key direction for future research is the development of resolution-adaptive processes that can effectively transfer learned features across varying image dimensions. Third, we only include 2D Gaussian mask for 2D sampling patterns during pretraining and downstream evaluations. More intensive evaluation across diverse sampling patterns (e.g., radial, spiral, or Poisson-disc) would provide further enhancement to the model’s robustness and generalizability. While our generalizability test from 1D to 2D masks demonstrates translational potential, we note that prospective 2D undersampling is primarily relevant for 3D imaging acquisitions. Thus, more intense work is needed to further validate its clinical applicability in the future. Fourth, our downstream tasks are primarily focused on image restoration. Recent works like MAE27 and SAM28 have demonstrated the versatility of pre-trained encoders for tasks such as classification and object detection. Future research directions could explore integrating MR-IPT into additional clinical workflows for tasks such as image segmentation and lesion detection, involving radiologists to further assess its diagnostic value. This extension would help evaluate the framework’s broader utility and applicability in medical imaging analysis beyond restoration tasks. Moreover, downstream interference tests (e.g., noise injection, adversarial attacks) after inference would provide further understanding regarding model’s resilience and applicability in real-world deployments. Additionally, integrating other advanced techniques such as denoising diffusion models38 and flow matching39,40 holds potential for developing more generalized multimodal frameworks in medical imaging. Integrating MR-IPT with these methods could broaden its applicability beyond MRI restoration and image processing. Finally, our MR-IPT currently employs a simple L1 loss function during training, which might limit its perceptual quality. Future studies will explore more sophisticated objective functions, such as SSIM loss and contrastive loss, to further enhance the performance.

In conclusion, we present MR-IPT, a ViT-based framework for general accelerated MRI restoration. MR-IPT demonstrates superior performance across various sampling setups, including new sampling masks and ratios, zero-shot learning, and limited dataset scenarios. By leveraging a multi-head-tail structure and a shared backbone design, MR-IPT exhibits strong adaptability to diverse acceleration configurations. Its efficient latent representation learning and large-scale pre-training capabilities highlight the potential of ViT-based architecture in advancing medical imaging deep learning models. This approach represents a significant step forward in the development of generalizable, high-performance deep learning models for medical imaging, with the potential to advance both clinical applications and future AI-driven diagnostic tools.

Methods

MR-IPT model structure and details

The MR-IPT framework builds upon a modified IPT structure, specifically designed to accommodate a wide range of accelerated MRI restoration tasks. The architecture of MR-IPT is illustrated in Fig. 1, showcasing its five core components: heads, tails, a prompt encoder, a shared encoder, and a lightweight decoder.

Each head consists of a 3 × 3 convolutional layer followed by two 5 × 5 residual blocks, enabling robust feature extraction from accelerated images. The tail, responsible for image restoration, includes a 3 × 3 upsampling convolutional block followed by another 3 × 3 convolutional layer to refine output quality. The prompt encoder, inspired by the design of the Segment Anything Model (SAM)28, utilizes sparse embeddings with an embedding layer to project acceleration information into the same dimensional space as image embeddings, enhancing task-specific feature representation.

For the shared encoder, we adopted a 24-layer ViT equipped with W-MSA34, which effectively captures global contextual information across multiple layers. This design mirrors the heavy-weight encoder/light-weight decoder paradigm seen in MAE27, balancing computational efficiency with strong representational capacity. The 2-layer lightweight decoder incorporates two-way cross-attention mechanisms between image patch tokens and prompt tokens, facilitating effective feature refinement and image restoration.

During pre-training, MR-IPT is exposed to diverse acceleration ratios (2×, 4×, 6×, 8×, and 10×) and a variety of sampling masks. The 1D sampling masks include Cartesian random, Cartesian equispaced, and 1D Gaussian, while the 2D sampling mask is based on 2D Gaussian distributions. We developed three MR-IPT variants to explore different aggregation strategies: (1) MR-IPT-type: Heads and tails are grouped based on acceleration ratios, with each head-tail pair specialized for different sampling masks; (2) MR-IPT-level: Heads and tails are aggregated based on sampling masks, allowing each pair to handle various acceleration ratios; (3) MR-IPT-split: A dedicated head-tail pair is assigned to each unique combination of sampling mask and acceleration ratio.

For downstream tasks, the appropriate head-tail pair is selected based on the specific sampling configuration. In cases involving unseen acceleration ratios (e.g., 5×), we utilize the head-tail pair trained for the nearest higher ratio (e.g., 6×). For MR-IPT-1D models, pre-training is limited to 1D masks, and during downstream evaluations, head-tails trained with Cartesian random masks are preferred due to their robustness against diverse sampling strategies.

MR-IPT was implemented using PyTorch41 and trained on systems equipped with either an NVIDIA RTX 3090 Ti (24GB VRAM) or an NVIDIA RTX 4090 (24GB VRAM). The model optimization follows an Adam optimizer42 with a learning rate of 1e-5. The training protocol includes a pre-training for 5 epochs on the large-scale dataset, then a fine-tuning for 15 epochs for each downstream restoration task.

The training objective for MR-IPT is L1 loss, defined as:

$$\:\begin{array}{c}L=\sum\:_{i=1}^{N}{\left|\left|MRIPT\left({x}_{accelerated}^{i}\right)-{x}_{clean}\right|\right|}_{1},\end{array}$$
(1)

where \(\:{x}_{accelerated}^{i}\) denotes the accelerated image for sampling task \(\:i\), and \(\:{x}_{clean}\) denotes the fully sampled clean ground truth image.

Datasets and image processing

For large-scale pre-training, we utilized the RadImageNet dataset35, which supports the findings of this study and is publicly available at https://www.radimagenet.com/. Specifically, we employed the MRI subset, comprising 672,675 images, which were split into a 9:1 ratio, resulting in 605,408 images for training and 67,267 for validation. For downstream task evaluations, we used the fastMRI dataset36, which is openly accessible at https://fastmri.med.nyu.edu/. For fastMRI knee dataset, we used the training set including 34,742 images for training. For testing, we used the validation set including 7135 images. For fastMRI brain dataset, we used the training set including 70,748 images for training. For testing, we used the test set including 8852 slices. To standardize the data, all images were resized to 224 × 224 pixels with pixel intensity values normalized within the range of [0, 1] based on the down-sampled image. Given the variability in image dimensions across the fastMRI datasets, we first applied center cropping to reduce the images to 320 × 320 pixels, ensuring the preservation of central anatomical features, and subsequently resized (downsampling interpolation) them to the target dimensions for model training and evaluation. To keep a fair comparison, for methods that requires k-space data as inputs such as E2E-VarNet and ReconFormer, the k-space data was first back-calculated from the resized fully sampled images. Then, this k-space data was undersmapled by corresponding masks and serve as inputs.

Evaluation metrics

To comprehensively assess restoration performance, we adopted two widely used quantitative metrics during our restoration performance evaluation comparisons: peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM). We report slice-wise PSNR/SSIM across our comparisons.

$$\:\begin{array}{c}PSNR\left(x,{x}_{clean}\right)=10{log}_{10}\frac{{\text{max}\left({x}_{clean}\right)}^{2}}{MSE\left(x,{x}_{clean}\right)},\:\end{array}$$
(2)

where \(\:MSE(x,{x}_{clean})\) is the mean squared error between the reconstructed image \(\:x\) and the fully sampled ground truth clean image \(\:{x}_{clean}\).

$$\:\begin{array}{c}SSIM\left(x,{x}_{clean}\right)=\frac{\left(2{\mu\:}_{x}{\mu\:}_{{x}_{clean}}+{C}_{1}\right)\left(2{\sigma\:}_{x{x}_{clean}}+{C}_{2}\right)}{\left({\mu\:}_{x}^{2}+{\mu\:}_{{x}_{clean}}^{2}+{C}_{1}\right)\left({\sigma\:}_{x}^{2}+{\sigma\:}_{{x}_{clean}}^{2}+{C}_{1}\right)},\:\end{array}$$
(3)

where \(\:{\mu\:}_{x}\), \(\:{\mu\:}_{{x}_{clean}}\), \(\:{\sigma\:}_{x}^{2}\), and \(\:{\sigma\:}_{{x}_{clean}}^{2}\) are the mean and variance of reconstructed image \(\:x\) and fully-sampled clean image \(\:{x}_{clean}\), respectively. \(\:{\sigma\:}_{x{x}_{clean}}\) is the covariance of \(\:x\) and \(\:{x}_{clean}\). \(\:{C}_{1}={\left(0.01L\right)}^{2}\), \(\:{C}_{2}={\left(0.03L\right)}^{2}\), where \(\:L\) is the dynamic range of the pixel-values.