Introduction

To acquire comprehensive angiographic information of both arterial and venous cerebral vasculatures in animal models, contrast-enhanced three-dimensional (3D) ultrashort echo time magnetic resonance angiography (UTE-MRA) with monocrystalline iron oxide nanoparticles (MION) is a promising approach1,2,3,4,5. The impressive resolution of this imaging technique plays a critical role in delineating the anatomical structures of fine blood vessels; however, achieving such high resolution comes at the cost of longer scanning durations6,7, which may introduce motion artifacts and, more critically, impact the promptness of diagnosis and treatment for time-sensitive conditions like stroke8,9. To illustrate, Fig. 1 depicts the imaging of ischemic stroke through diffusion imaging (B and C) and angiographic methods (D and E). Notably, the smaller vasculatures exhibit enhanced visibility in the high-resolution (HR) contrast-enhanced UTE-MRA image (E), albeit requiring a fourfold increase in acquisition time.

Fig. 1
figure 1

Imaging of ischemic stroke. (A) Schematic of the MCAO model. (B) Diffusion-weighted image (DWI, axial view), and (C) trace map of the apparent diffusion coefficient (ADC) in µm2/s (axial view). (D) and (E) are coronal views of low- and high-resolution UTE-MRA data, respectively, after maximum intensity projection. The MCAO model schematic was generated by overlaying cerebral vasculature on a brain template. The hyperintensity in the DWI and hypointensity in the ADC map show the region of infarction. MCAO: middle cerebral artery occlusion.

Full size image

Currently, deep learning has captured considerable interest due to its exceptional performance in diverse computer vision tasks. These include image classification, segmentation, localization, and the generation of super-resolution (SR) images, a capability that classical algorithms previously struggled to achieve9,10,11,12. As obtaining HR 3D MRA data poses considerable challenges, employing a convolutional neural network (CNN) to generate a SR angiogram from a quickly acquired low-resolution (LR) angiogram presents a promising solution. This approach holds potential for enhancing the sensitivity and specificity of angiographic investigations, ensuring model stability, and expanding preclinical availability9,10.

Based on the training strategy, single-image SR CNNs can be broadly categorized into peak signal-to-noise ratio (PSNR), flow, generative adversarial network (GAN), and diffusion-driven approaches11,12. PSNR-driven models are easier to implement and train but struggle to recover high-frequency edge detail in images. On the other hand, flow-based models exhibit a substantial computational footprint11. It is widely acknowledged that GAN-based models, due to their adversarial nature, encounter challenges in training and are susceptible to various failure modes9,10,11. Furthermore, even when these models are successfully trained, they tend to produce artifacts that appear natural, posing a significant challenge for their application in the medical field. Diffusion-based SR models are the most recent, which use LR data and stochastic noise as input to generate SR images11. While these models exhibit stability, their processing speed is notably slower compared to alternative methods. But it is also worth noting that, from a single LR image, the generation of multiple SR images is possible based on another input, introducing an element of randomness11. In this research, PSNR, as well as a combination of PSNR and GAN-oriented training, were implemented.

Beyond training methodologies, numerous researchers have put forth varied network architectures. The SR residual network (SR-ResNet) stands out as a pioneering example, utilizing residual blocks for feature extraction along with diverse up-sampling methods including interpolation-based techniques, inverse convolution, and pixel shuffling10,13. The memory-efficient residual dense generator (MRDG) enhances SR-ResNet by substituting residual blocks with residual dense blocks, which results in improved performance. However, this enhancement comes at the expense of increased computational demands and prolonged inference times9. For handling 3D and time-sensitive medical data, efficient deep learning models with reduced computational costs and inference times are crucial9,10,11. In this study, 3D SR-ResNet, MRDG64, and the newly proposed model were implemented and a comparative study was performed. The objective of the current study is to enhance the resolution of UTE-MRA data acquired within a short time frame by leveraging a tailored combination of CNN models, cost function, and training strategy. This refinement aims to position UTE-MRA as a more effective alternative in diagnosing conditions such as stroke and other vascular diseases, especially in animal models.

Methods

Data

Animal preparation

Adult male and female Wistar rats (body weight 250–350 g) were kept in cages with free access to food and water, housed inside a room with a 12-h light/12-h dark cycle. All animal experiments were carried out after receiving approval of animal care by the Institutional Animal Care and Use Committee (IACUC, IACUC-23–25) of the Ulsan National Institute of Science and Technology (UNIST) and carried out in accordance with the Animal Protection Act of Korea and Guide for the Care and Use of Laboratory Animals of the National Institute of Health. All the animal experiments in this study are reported in accordance with ARRIVE guidelines. Animals were anesthetized using 1.5–3% isoflurane mixed with 30% and 70% oxygen and nitrous oxide, respectively, throughout the experiment based on the breathing rate, which was kept at 30–45 breaths/minute. In addition, during the MRI experiments, the body temperature of the animals was kept constant (37 ± 1 °C) via the circulation of warm water through the animal bed. Ischemic stroke was induced by permanent occlusion of the left middle cerebral artery (MCA) and the acquisition of MRI data immediately after surgery14. An appropriately sized silicone rubber-coated monofilament (Doccol Corporation, Redlands, CA, USA) was inserted through a tiny hole in the common carotid artery (CCA) to maintain MCA occlusion throughout the experiment5,15.

MRI Acquisition

A total of 20 paired contrast-enhanced 3D UTE-MRA data were collected from healthy control (HC) animals using the UTE pulse sequence with an isometric matrix size of 1283 (LR) and 2563 (HR) with a voxel size of 2343 μm3 and 1173 μm3, respectively, time of repetition (TR) of 22 ms, and echo time (TE) of 0.012 ms after the intravenous bolus injection of MION at a dose of 360 μmol/kg5. The scanning times for HR and LR data were 66 and 16 min, respectively. Diffusion-weighted images (DWIs) were acquired with b = 1000 s/mm2, three orthogonal diffusion directions, matrix size of 128 × 128, and field of view of 30 × 30 mm2, and used only to evaluate the induction of stroke using an apparent diffusion coefficient (ADC) map, as shown in Fig. 1(C). Additionally, a total of 10 paired 3D UTE-MRA data were collected from animals with ischemic stroke using the same protocol. Both DWI and UTE-MRA data were acquired using a 7 T MR scanner (Bruker, Ettlingen, Germany) and a 40 mm volume coil. The raw MRA data were preprocessed and registered to reduce noise and motion artifacts, respectively16. To facilitate computation and achieve better performance, the original LR and HR data were fragmented into 323 and 643 patches, respectively, with 50% overlap between successive patches during training.

Convolutional neural networks

In this research, a new 3D single-image super-resolution generative network named the ladder-shaped residual dense generator (LSRDG) was designed and implemented using residual dense blocks. To our knowledge, all previously developed SR image generators up-sample the high-level features only at the distal portion of the network before the last convolution, while on the other hand, LSRDG up-samples features after each residual dense block. The up-sampled features from the previous layer are passed through a simple convolution layer and added to the next up-sampled features, thereby forming a ladder-shaped network as shown in Fig. 2. The network forms multiple routes or skip connections for the gradient of the loss from the output back to the input layer, which helps the network to be numerically stable and less prone to overfitting as well as to achieve increased performance17,18,19. As the convolution operation on high spatial features is computationally expensive, simple convolution followed by activation was used on the HR features track instead of the residual dense blocks in the LR features track of the network. Each building block of the model was included through extensive experiments to increase the generative ability of the model with time as a constraint. LSRDG was effectively optimized to increase performance, reduce the computational footprint, and reduce inference time using a feature depth of 32, kernel size of 3, dense block growth rate of 16, and 6 residual dense blocks with Gaussian error linear unit (GELU) activation20. During our ablation experiment, it was noted that adding more blocks, increasing the number of filters from 32 to 64, or using residual dense block instead of a single convolution layer in the HR feature of the model increases the performance with a longer inference time and higher computational cost. In addition to the architecture of the proposed model, different training strategies such as PSNR only and a combination of PSNR and GAN oriented loss were implemented and compared. The GAN version of our model, called the ladder-shaped residual dense generative adversarial network (LSRDGAN), uses a pyramid pooling discriminator with instance normalization and leaky rectified linear unit (ReLU) activation function, allowing the network to discriminate images at different image resolutions9.

Fig. 2
figure 2

Ladder-shaped residual dense generator (LSRDG), the proposed network architecture.

Full size image

For comparison to the developed method, other networks were additionally implemented in this work as follows. The first was 3D SR-ResNet with 6 residual blocks and batch normalization, according to Sánchez et al.10 A 3D super-resolution generative adversarial network (SRGAN) was also implemented using SR-ResNet as a generator, and a visual geometry group (VGG)-styled discriminator with the last fully connected classifier without a hidden layer to stabilize the training and be less prone to GAN failures10. The other network was MRDG64, implemented according to Wang et al., with 64 residual features, a dense block growth rate of 12, and 6 memory-efficient residual dense blocks each having 3 residual dense blocks9. Similar to SRGAN, an enhanced super-resolution generative adversarial network (ESRGAN) was also implemented using MRDG64 as a generator and pyramid pooling discriminator9. In all models, 3D up-sampling based on nearest neighbor interpolation was used, which has the lowest computational cost and better performance compared with transposed convolution and pixel shuffling methods10. For the binary classification task of the discriminator, soft labeling with decay was applied instead of hard labeling to suppress overconfidence and achieve stable training of the GANs as shown in Eq. 121,22.

All networks were first optimized to find the best hyperparameters. PSNR-oriented and the corresponding PSNR and GAN–oriented networks were trained using L2 loss and a combination of L2 and adversarial loss with a scaling factor set to 0.001, as shown in Eqs. 2 and 3, respectively. L2 loss was computed on the image as well as on the gradient of HR and SR images to recover the high-frequency components. The Nesterov-accelerated adaptive moment estimation (NADAM) optimizer, a batch of 16 patches, and an initial learning rate of 1e-4 that decays step-wise by half every 100 epochs were used23. The maximum number of epochs was set to 1000 and the training was halted before overfitting by evaluating the validation loss. All models were implemented in PyTorch (version 1.13.1, Meta AI, Menlo Park, CA, USA) and trained using a Nvidia A100 tensor core 80 GB graphics processing unit (Nvidia Corporation, Santa Clara, CA, USA). All networks were trained on 5145 patches, validated using 1715 patches from 15 and 5 HC experiments respectively using k-fold cross-validation, and then tested on 10 ischemic stroke data. To compare the network performances in generating SR images from the corresponding LR images, structural similarity index measure (SSIM), PSNR, and mean squared error (MSE) were used as evaluation metrics in addition to visual comparison. Lastly, the performance of our proposed model with and without GAN on HC and stroke data was compared with other models using the Mann–Whitney U test with statistical significance set at P < 0.05.

Loss equations

Discriminator loss

$$Binary \ cross \ entropy (BCE) \ loss= - \frac{1}{N}\sum_{i=1}^{N}{{y}_{i}}{\prime}*\text{log}\left({p}_{i}\right)+(1-{{y}_{i}}{\prime})*\text{log}\left(1-{p}_{i}\right)$$
(1)

where \({y}{\prime}\) is the soft label with the decay of HR and SR images.

Generator loss

$$PSNR \ loss = L_{2} \left( {I^{HR} , I^{SR} } \right) + L_{2} \left( {\nabla I^{HR} , \nabla I^{SR} } \right) = \frac{1}{{N^{3} }}\mathop \sum \limits_{x = 1}^{N} \mathop \sum \limits_{y = 1}^{N} \mathop \sum \limits_{z = 1}^{N} \left( {I_{xyz}^{HR} - I_{xyz}^{SR} } \right)^{2} + \frac{1}{{\left( {N - 1} \right)^{3} }}\mathop \sum \limits_{x = 1}^{N - 1} \mathop \sum \limits_{y = 1}^{N - 1} \mathop \sum \limits_{z = 1}^{N - 1} \left( {\nabla I_{xyz}^{HR} - \nabla I_{xyz}^{SR} } \right)^{2}$$
(2)
$$Generator\,loss = PSNR \ loss - \frac{\alpha }{N}\sum\limits_{i = 1}^{N} 1 {^{\prime}}*{\text{log(}}{p}_{i}{)} + (1 - 1{^{\prime}})*{\text{log(1}} - {p}_{i}{)}$$
(3)

where \(\nabla\) is the gradient operator of the image along three axes, N is the dimension of the image (height = width = depth = N = 64), α is the scaling factor of the adversarial loss, and \({1}{\prime}\) is the soft label with decay corresponding to 1 in hard labeling.

Results

Table 1 summarizes the properties of the SR image generator models in terms of the number of parameters, number of floating-point operations (FLOPs), and inference time. Compared to the MRDG64 model, our proposed generative model has a significantly lower number of parameters (~ 1.42 M), and computational footprint (5.5 TFLOPs) while taking less time (~ 18 s) to transform LR into SR. Among all models, SR-ResNet has the smallest number of parameters and lowest computational cost as well as the shortest inference time.

Table 1 Comparison of generator networks using the number of parameters, inference time, and number of floating-point operations (FLOPs).
Full size table

The performances of the networks trained with and without GAN were compared using SSIM, PSNR, and MSE as shown in Tables 2 and 3. Our proposed model trained both with and without GAN significantly outperforms the other models in all evaluation metrics and reconstructs an image that is similar to the original HR image. Figures 3 and 4 show a visual comparison of SR images and residual maps respectively generated using PSNR-only oriented models on HC data. In addition, Fig. 5 shows the visual comparison of generated SR images using a combination of PSNR and GAN-oriented models on HC data. As indicated by the arrows in Figs. 3 and 5, the bifurcation of blood vessels was visualized better using our proposed model compared to others in both modes of training.

Table 2 Statistical comparison of the performance of generator networks on HC data.
Full size table
Table 3 Statistical comparison of the performance of generator networks on stroke data.
Full size table
Fig. 3
figure 3

Visual comparison of generated images using PSNR-oriented training on HC data. The first row shows images after maximum intensity projection in the coronal direction. The second row shows the focused region (marked by the red square in the top row), and the third row shows the volume-rendered data. The yellow and red arrows on the images show the bifurcation points of the blood vessels that are successfully reconstructed by our model only (yellow arrows) and not reconstructed by any model (red arrows).

Full size image
Fig. 4
figure 4

Visual comparison of absolute residual maps generated using PSNR-oriented training from HC data. The first row shows original HR images along with generated SR images using cubic spline interpolation and PSNR-oriented models after maximum intensity projection in the coronal direction. The second row shows the absolute difference map between the original data and the generated SR image and the third row shows the difference map of the focused region (marked by the red square in the top row).

Full size image
Fig. 5
figure 5

Visual comparison of generated images using PSNR and GAN–oriented models on HC data. The first row shows images after maximum intensity projection in the coronal direction. The second row shows the focused region (marked by the red square in the top row), and the third row shows the volume-rendered data. The yellow and red arrows on the images show the bifurcation points of the blood vessels that are successfully reconstructed by our model only (yellow arrows) and not reconstructed by any model (red arrows).

Full size image

In addition, the proposed model trained without and with GAN also performed better on stroke data compared to others and generated a very similar image to the original HR image, as shown in Figs. 6 and 7, respectively. For further comparison, Supplementary Figs. 1 and 2 as well as Supplementary Animations 1 and 2 show generated images using PSNR-oriented models and PSNR and GAN–oriented models on HC data, respectively. In addition, Supplementary Figs. 3 and 4 as well as Supplementary Animation 3 also show a visual comparison of different models trained on HC data and applied to stroke data. In general, compared with the cubic spline interpolation, all deep learning models performed significantly better on both HC and stroke data.

Fig. 6
figure 6

Testing PSNR-oriented models on stroke data. The first row shows the HR and LR along with the SR data generated using PSNR-oriented models after maximum intensity projection in the sagittal direction. The second row shows volume-rendered data. The yellow ellipse on the original image shows the region of infarction due to ischemic stroke, and the yellow arrows show the visual difference of the vessel in the contralateral hemisphere, which appears occluded in the LR image.

Full size image
Fig. 7
figure 7

Testing PSNR and GAN–oriented models on stroke data. The first row shows the HR and LR along with the SR data generated using PSNR and GAN–oriented models after maximum intensity projection in the sagittal direction. The second row shows volume-rendered data. The yellow ellipse on the original image shows the region of infarction due to ischemic stroke, and the yellow arrows show the visual difference of the vessel in the contralateral hemisphere, which appears occluded in the LR image.

Full size image

Average SSIM, PSNR, and MSE scores of 0.983, 36.80, and 0.00021, respectively, were achieved using our proposed model trained with PSNR-oriented loss compared to 0.964, 34.38, and 0.00037 using SR-ResNet and 0.978, 35.47, and 0.00029 using MRDG64 on HC data. In addition, when trained only on HC data and tested on stroke data, our model achieved SSIM, PSNR, and MSE scores of 0.963, 34.14, and 0.00041, respectively, compared to 0.953, 32.24, and 0.00061 using SR-ResNet and 0.957, 32.90, and 0.00054 using MRDG64. The image quality of LSRDG was also greatly improved compared to up-sampling using cubic spline interpolation, achieving SSIM, PSNR, and MSE scores of 0.848, 27.61, and 0.0019 on HC data and 0.831, 26.64, and 0.0025 on stroke data, respectively.

Statistically, our proposed PSNR-only driven model showed a significant improvement in terms of SSIM, PSNR, and MSE on HC data (all p < 10–5) and stroke data (p = 0.003, p < 10–4, and p < 10–4) compared with SR-ResNet. Similarly, it also achieved significant improvement for the same metrics on HC data (p = 0.0002, p < 10–4, and p < 10–4) and stroke data (p = 0.048, p = 0.0005, and p = 0.0005) compared with MRDG64. When trained with PSNR and GAN loss, LSRDGAN showed significant improvement on HC data compared with SRGAN (all p < 10–6) and with ESRGAN (p = 0.002, p = 0.007, and p = 0.006) in terms of SSIM, PSNR, and MSE, respectively. In addition, we found improvement compared with SRGAN on stroke data (p = 0.007, p < 10–4, and p < 10–4) with our model for SSIM, PSNR, and MSE, respectively. But while our proposed GAN-driven model provided better performance on stroke data compared with ESRGAN, statistically it was not significant in terms of the three respective metrics (p = 0.05, p = 0.240, and p = 0.281).

Discussion and conclusion

SR methods have been commonly applied to different categories of medical images8,9,10,24,25,26, but less so on 3D angiographic data. Previously, both 2D and 3D deep learning–based SR networks were applied to clinical MRA data to up-sample LR images generated from HR images by sub-sampling in k-space by a factor from 2 to 6 times, showing the possibility of reducing the scanning time by a factor of four27. Recently, the resolution of clinical time-of-flight magnetic resonance angiography (TOF-MRA) images was improved using a 2D-based pix2pix GAN, which led to significant improvements in diagnostic performance compared with LR TOF-MRA28. In previous works with the aim to increase the resolution of MRI data using deep learning, LR data were frequently synthetically generated by downgrading HR data in k-space or the image domain9,10,27. Comparing synthetically generated LR data and actual LR data from MR experiments, the noise characteristics may be different, which can create data mismatches and degrade performance when applying a model trained on synthetic LR data to real LR data. Specifically, UTE-MRA utilizes non-uniform radial sampling that must be re-grid to Cartesian format before applying a Fourier transform, and the optimum acquisition as well as reconstruction parameters may not be the same for both HR and LR data, resulting in additional sources of data mismatch29,30. To our knowledge, this is the first study to use contrast-enhanced UTE-MRA data to enable consistent registration between LR- (not synthesized) and HR-acquired 3D whole-brain angiograms.

The quality of a generated SR image is strongly influenced by numerous factors such as the model architecture, training data, optimization methods, and the chosen cost function. In comparison to SR-ResNet and MRDG64, our proposed model consistently produced superior SR images with less artifacts when applied to UTE-MRA data. Enhancing the network’s ability to generate more accurate data involves up-sampling features at different depths and connecting them after simple convolution. Although increasing the depth of the network leads to improved performance, it comes with increased computational cost and inference time. Given our goal of reducing the overall time to obtain SR data, we opted for a fixed depth of 6 residual dense blocks. In addition, training SR models using PSNR-oriented loss, such as L2 loss, tends to generate smoothed images, requiring different methods to be employed to recover the edge details of the images. On the other hand, while GAN loss exhibited the capability to generate both high- and low-frequency components of the image, it faced challenges related to artifacts and stability during training28. Since medical data are highly sensitive and artifacts may lead to misdiagnosis, training the model with a cost function that is sensitive enough for both contrast and edge detail and that generates less artifacts is considered appropriate. Notably, employing a smaller patch size and incorporating L2 loss on both the image and gradient of HR and SR significantly enhances the image quality10,31.

As MRA in general uses flow-induced contrast, it is well known that we can’t see the blood vessels that are fully occluded due to the absence of flow even in the original HR images32,33. Our work is exclusively focused on better visualization of healthy blood vessels which can be seen by UTE-MRA in a shorter time and using a less computationally intensive model. The LR data acquisition takes 16 min, only a quarter of the time needed for obtaining a HR image. But LR images lack details of smaller vasculatures that are clearly visible in HR images. Figures 6 and 7 as well as Supplementary Figs. 3 and 4 highlight the similarity between the ipsilateral and contralateral hemispheres in a LR image, with some blood vessels appearing occluded even in the contralateral region (indicated by the arrows in Figs. 6 and 7). This discrepancy poses the risk of causing confusion and misguided diagnosis and treatment. In contrast, such disparities are noticeably absent in both HR and SR images. Despite not being trained with stroke data, our model produced SR images that closely resembled the HR images, achieving this with a significantly reduced time (16 min acquisition and 18 s inference time). This approach demonstrates the potential for training deep learning models on HC data and subsequently applying them to pathological condition data. This is especially pertinent in cases where obtaining HR data from patients with time-sensitive diseases, such as chronic stroke, is clinically infeasible.

The quantity of training data in this research was limited, a constraint we mitigated through the utilization of k-fold cross-validation with acknowledgment of the well-established reliance of deep learning models on data abundance. In addition, likely due to the presence of dynamic vascular changes in the permanent stroke model during the acquisition of both HR and LR data, all methods showed lower performance compared with HC data, an effect that can be lessened by including stroke data in the training dataset. Due to the absence of a completely accurate SR model, it remains challenging to reconstruct all smaller blood vessels and bifurcations that are visible on an original HR image, as shown by the red arrows in Figs. 3 and 5, which can be improved by training the models on a larger dataset and applying further optimizations28. But despite the challenges to recover all image details of HR, the improvement is significant compared to using LR images9,10,27,28,34,35. Therefore, the proposed approach may be clinically advantageous if applied to time-sensitive cases where acquiring HR images is technically challenging or risky28,35. In addition, considering stroke and non-cooperative patients, 16 min may not be clinically feasible, but if clinically translated, the bigger blood vessels of humans than those of rodents may make it possible to start from lower resolutions, which takes a fraction of the time based on the matrix size. Upon clinical translation, this approach can help both clinicians as well as patients in reducing time and cost, thereby increasing patient safety as well as improving diagnostic accuracy of patients with different vascular malfunctions such as ischemia, hemorrhage, aneurysm, and others. Lastly, by synergistically combining with compressed sensing and faster imaging techniques, SR models using deep learning can greatly help to speed up acquisition and minimize the main limitation of MRI in general.

In conclusion, the resolution of contrast-enhanced UTE-MRA was significantly improved compared to LR by combining a well-designed network, suitable loss function, and training with a smaller patch size. Our proposed PSNR-only driven model was able to outperform other methods and provided SR data that is close to the desired HR data.