A deep learning framework for instrument-to-instrument translation of solar observation data

Abstract

The constant improvement of astronomical instrumentation provides the foundation for scientific discoveries. In general, these improvements have only implications forward in time, while previous observations do not benefit from this trend, and the joint use of data sets from different instruments is typically limited by differences in calibration and quality. We provide a deep learning framework for Instrument-To-Instrument translation of solar observation data, enabling homogenized data series of multi-instrument data sets. This is achieved by unpaired domain translations with Generative Adversarial Networks, which eliminate the need for spatial or temporal overlap to relate instruments. We demonstrate that the available data sets can directly profit from instrumental improvements, by applying our method to four different applications of ground- and space-based solar observations. We obtain a homogenized data series of 24 years of space-based observations of the solar EUV corona and line-of-sight magnetic field, solar full-disk observations with increased spatial resolution, real-time mitigation of atmospheric degradations in ground-based observations, and unsigned magnetic field estimates from the solar far-side based on EUV imagery. The direct comparison to simultaneous high-quality observations shows that our method produces images that are perceptually similar, and enables more homogeneous multi-instrument data sets without the requirement of spatial or temporal alignment.

Introduction

With the rapid improvement of space-based and ground-based solar observations, highly resolved details of the solar surface and atmospheric layers have been obtained. As compared to the 11-year solar activity cycle the development of instruments progresses over smaller time scales. This imposes additional challenges for the study of long-term variations and combined usage of different instruments. In this study, we address the question on how the additional information obtained from instrumental improvements can be utilized to enhance observations of lower quality. This especially aims at homogenizing long time-series, mitigating atmospheric influences and to overcome instrumental limitations (e.g., resolution limitation, seeing).

The automated homogenization of data sets provides an integral component for long-term studies (e.g., solar cycle studies, historic sunspots, studies of rare events) and for studies that combine data from multiple instruments. Especially when dealing with large amounts of data, the automatic adjustment can be faster and more consistent than treating the data sets of the individual instruments separately^1,2,3. Data driven methods rely on the diversity and amount of data, and the inclusion of additional data sets can significantly increase the performance⁴. Enhancing old observation series to the standard and quality of the primary modern data set can provide an easily accessible data source. Methods developed for specific instruments often depend on certain observables (e.g., magnetograms, filtergrams of specific wavelengths) that are only partially covered by other data sets. An approximation based on proxies can already provide a suitable basis for automated methods or gives additional information for solar monitoring (e.g., in the frame of space weather predictions). In the regular observation schedule the mitigation of atmospheric effects and quality enhancement is a frequently addressed problem^5,6,7.

A principle problem of every enhancement method is the absence of a reference high-quality image. The inversion of the image degradation (e.g., lower spatial resolution, instrumental characteristics) is therefore inferred from artificial degradations^8,9,10, from simulation data^11,12,13, by estimating the degrading effects^14,15, or by co-alignment of different data sets¹⁶. We argue that a designed degradation can only represent a limited set of observations and can not account for the full diversity of the actual quality decreasing effects that occur in solar observations. Especially when dealing with atmospheric effects (i.e., clouds, seeing) and instrumental characteristics, the quality degradation is complex to model^17,18. Even with the precise knowledge of the degrading function, every image enhancement problem is ill-posed^19,20. However, it’s possible to reduce the number of possible high-quality solutions for a given low-quality image significantly by considering the distribution of real high-quality images and requiring that enhanced images correspond to the same image domain.

In this study, we propose an approach that uses real observations from state-of-the-art instruments in observational solar physics, to model the image quality distribution of high-quality images. We overcome the limitation of a high-quality reference image with the use of unpaired image-to-image translation²¹. We provide a general framework that translates from a given low-quality domain to a target high-quality domain (Instrument-To-Instrument translation; ITI). With this approach, we infer information from real observations to enhance physically relevant features which are otherwise beyond the diffraction limit of the telescope (e.g., super resolution), inter-calibrate data sets, mitigate atmospheric degradation effects and estimate observables that are not covered by the instrument.

Results

Our primary model architecture consists of two convolutional neural networks (generator BA and generator AB), where the first generates synthetic low-quality images from a given high-quality image (generator BA). The second network is trained to invert the image degradation to reconstruct the original high-quality observation (generator AB). We enforce the generation of low-quality images with the use of competitive training between generator BA and a discriminator network. We include an additional noise factor for generator BA to model a variety of degrading effects, independent of the image content^22,23. With the synthesis of more realistic and diverse low-quality observations, the generator AB is capable to provide a similar reconstruction performance for real low-quality observations (Fig. 1). The artificial degradation leads inevitably to an information loss that needs to be compensated by the generator AB to reconstruct the original image. Analogously to the training cycle in Fig. 1 we employ a cycle translating low-quality observations to high-quality observations (A-B-A, see “Methods” section). This enforces that images by generator AB correspond to the domain of high-quality images, restricting the possible enhanced solutions and gaining information from the high-quality image distribution.

Fig. 1: Overview of the model training cycle for the synthesis of low-quality images.

Images are transformed from the high-quality domain (B) to the low-quality domain (A) by generator G BA (yellow). The synthetic images are translated by generator G AB (blue) back to domain B. The mapping into domain A is enforced by discriminator D A, which is trained to distinguish between real images of domain A (bottom) and generated images (top). Both generators are trained jointly to fulfill the cycle consistency between original and reconstructed image, as well as for the generation of synthetic images that correspond to domain A. The generation of multiple low-quality versions from a single high-quality image is accomplished with the additional noise term that is added to generator BA.

In contrast to paired image translation tasks in solar physics^{1,10,16,24,25,26,27,28,29,30,31}, our method requires no alignment of the data sets. This allows for the translation between instruments that have no joint observing periods (e.g., historic data, different observing times, differences in cadence), or have differences in their spatial alignment (e.g., different field-of-view, observations from different vantage points in the heliosphere). Therefore, unpaired translations enable a much wider range of applications, specifically providing translations between different instruments, which are not feasible or would require extensive pre-processing in a paired translation setting.

In this study, we apply our method to four data set pairs and assess the validity of the enhanced images with the sparse set of simultaneous observations. The translation between data sets is performed between unpaired imaging data from different instruments that share similar observables (e.g., narrow-band filtergrams, continuum observations, magnetic field maps). For all applications, we distinguish between high- and low-quality data sets. Hereby we consider data set pairs, where the high-quality data set contains observations with a better spatial resolution or less quality degradations (e.g., noise, atmospheric effects), as compared to the low-quality data set. We consider four translation tasks:

• 1. Inter-calibration of extreme ultraviolet (EUV) filtergrams, using observations from the extreme ultraviolet imager (EUVI³²) onboard the solar terrestrial relations observatory (STEREO³³), the extreme-ultraviolet imaging telescope (EIT³⁴) onboard the solar and heliospheric observatory (SOHO³⁵), and the atmospheric imaging assembly (AIA³⁶) onboard the solar dynamics observatory (SDO³⁷). This also includes the inter-calibration of magnetograms from the the Helioseismic and Magnetic Imager (HMI³⁸) and the michelson doppler imager (MDI³⁹), onboard SDO and SOHO, respectively.

• 2. Image super-resolution of continuum observations from HMI by using observations from the solar optical telescope (SOT⁴⁰) onboard the Hinode mission. Specifically, we use red continuum observations from the broadband filter imager (BFI) from Hinode/SOT.

• 3. Mitigation of atmospheric degradations in ground-based Hα observations from the Kanzelhöhe Observatory for Solar and Environmental Research (KSO^41,42).

• 4. Estimation of the far-side solar magnetic field based on EUV observations from EUVI onboard the STEREO twin-satellites and magnetograms from HMI onboard SDO.

With this multi-application approach, we aim to provide a thorough evaluation to investigate the performance of the method in terms of instrument inter-calibration (1), physical validity (1), super-resolution of features (2), mitigation of noise (3), and approximation of observables (4). Furthermore, the applications are selected to demonstrate the applicability for data set pairs that have no temporal or spatial overlap (1, 2, 4), cover a different field of view (2), originate from the same instrument with varying quality (3), and differ in observables (4).

Throughout this study we apply a strict temporal split of the training data set (see, “Methods” subsection “Datasets”), to exclude a potential memorization of observations. We evaluate individual applications in detail and present quantitative metrics for paired samples across all applications in the subsection “Quantitative evaluation”.

Inter-calibration of multi-instrument data—SOHO/EIT + MDI and STEREO/EUVI to SDO/AIA + HMI

With the translation between image domains our method can account for both image enhancement and adjustment of the instrumental characteristics simultaneously (cf.⁴³). We use ITI to enhance full-disk EUV filtergrams from SOHO/EIT and STEREO/EUVI to SDO/AIA quality and calibrate the images into a unified series dating back to 1996.

With a spatial sampling of 0.6 arcsec pixels, we consider the AIA filtergrams and HMI magnetograms as high-quality reference. We use the EUV filtergrams from STEREO/EUVI as low-quality data set. For the translation of SOHO observations we use EIT filtergrams in combination with the LOS magnetograms of MDI. The SDO/AIA images provide a 2.7 times higher pixel resolution than the STEREO/EUVI images. We reduce the resolution of STEREO observations to 1024 × 1024 pixels, before using ITI to increase the resolution by a factor 4, to the full 4096 × 4096 pixels resolution of SDO (STEREO/EUVI-to-SDO/AIA). For observations from SOHO/EIT + MDI we use half of the SDO resolution as reference (SOHO/EIT + MDI-to-SDO/AIA + HMI). All observations are normalized to a fixed scale of the solar radius, to avoid variations due to the ecliptic orbits (see, methods subsection “Data pre-processing”). We consider all matching filters (131 Å, 195/193 Å, 284/211 Å, 304 Å, LOS magnetogram) and translate the combined set of channels with ITI, to benefit from the inter-channel relation.

The SOHO mission provides observations that partially overlap with observations from SDO. In Fig. 2b, we provide a side-by-side comparison between the original SOHO, enhanced ITI and reference SDO observations. The full-disk images are a composite of the ITI enhanced and SDO observations, which show the same calibration effect as the STEREO maps (Fig. 2a). The sub-frames show the four EUV filtergrams and the LOS magnetogram of different regions of interest. We show examples of a filament, a quiet-Sun region, the solar limb and an active region. For all samples we note a strong resolution increase and high similarity to the real reference observations. The filament in the top row, is difficult to observe in the original SOHO observation, but can be clearly identified in the ITI image. For the frame at the solar-limb in the 284/211 Å channel, we also note an inference from the multi-channel information that reconstructs the faint off-limb loops. From the quiet-Sun region (second row) and the full-disk images, we can see that our method is consistent across the full solar disk. The observation of the 304 Å channel (active region; fourth row) shows the strongest improvement as compared to the original observation, but also shows that smaller features could not be fully reconstructed. We note that pixel errors can lead to wrong translations and can be accounted for prior to the translation (e.g., noise in SOHO/EIT 284 Å). The magnetic elements in the LOS magnetogram of ITI show an improvement in sharpness and the reconstructed shape of the sunspot matches the HMI reference, but the full HMI quality cannot be reached. The magnetic flux elements at the lowest resolution level were not resolved by ITI. This shows the limit of the method, but also suggests that no artificial features are added by our model.

Fig. 2: Evaluation of the ITI translation for the homogenization of SOHO with SDO observations and calibration of EUV data series.

a Two examples of SOHO-to-SDO translation from 2010-05-13 07:00 (left) and 2010-08-05 01:00 (right). The boxes indicate the cutouts in panel b where we compare ITI images with aligned SDO/AIA filtergrams. We compare 300″ × 300″ and 100″ × 100″ cutouts of EUV filtergrams and LOS magnetograms, respectively. We directly compare observations taken at the same time from SOHO (left), ITI (center) and SDO (right). 131 Å: The observed feature is difficult to identify in the SOHO observation, while the ITI enhanced version resolves a clear filament structure that is consistent with the SDO observation. 195/193 Å: details in the quiet-Sun region are blurred in the SOHO image. The obtained features by ITI are consistent on a global scale with SDO, but more deviations occur at the smallest resolution scales. 284/211 Å: ITI recovers faint off-limb loops that are not resolved by SOHO. The pixel-noise at the bottom left is mitigated, but results in spurious features. 304 Å: The active region shows a valid reconstruction from the strongly pixelated observation. LOS Magnetogram: Small magnetic elements are better resolved and appear deconvolved in the ITI image. The shape of the sunspot is well reconstructed, however the full quality of the SDO/HMI magnetograms can not be reached by ITI. c Comparison of the SDO reference and calibrated SOHO/EIT and STEREO/EUVI EUV light-curves. The mean intensities for each channel are plotted against time in the individual panels. The data is smoothed by a running average filter, where the blue shaded area corresponds to 1 standard deviation of the SDO intensities. The solar cycle trend can be seen for each light-curve⁸⁷. ITI adjusts the individual observations to a similar scale (DN/s), which outperforms the baseline approach.

For the usage of long-term data sets the consistent calibration is of major importance^18,44. Data sets that comprise multi-instrument data require an adjustment into a uniform series^2,45,46. We apply our ITI method to the full observation series from STEREO/EUVI and SOHO/EIT, to achieve a homogenized data series with SDO/AIA. We evaluate the model performance for long-term consistency over more than two solar cycles by computing the mean intensity per channel and instrument and comparing the resulting light-curves, where we use a running average with a window size of one month to mitigate effects of the different positions of the SDO and STEREO satellites (Fig. 2c). We compare our method to a baseline approach where we calibrate the EUV observations based on the quiet-Sun regions (see, “Methods” subsection “Calibration”). As can be seen from Fig. 2c, our model correctly scales the SOHO/EIT and STEREO/EUVI observations to the SDO intensity scale, with a good overlap that suggest also valid calibrations for pre-SDO times. At the end of the nominal operation of SOHO a sudden increase in multiple channels can be noticed (e.g., 304 Å), this change in calibration restricts further assessment. Prior to this intensity jump, we note a good agreement between ITI and SDO/AIA 304 Å. In Table 1 we quantify the differences to the SDO reference in terms of mean-absolute-error (MAE) and cross-correlation (CC). For the STEREO observations our method shows throughout a higher performance, especially for the coronal EUV emission lines (171 Å, 195 Å, 284 Å). The SOHO comparison covers only simultaneous observations, which were recorded during solar minimum. Consequently, the baseline shows little deviations, and similar results are achieved by ITI.

Table 1 Evaluation of the intercalibration of SOHO, STEREO and SDO observations

In the time between March 2010 and April 2011, magnetograms of both SOHO/MDI and SDO/HMI can be directly compared. Here we compare the calibration between MDI and HMI. As can be seen from Fig. 3b the MDI instrument has an offset to the measured SDO/HMI flux. In addition, fluctuations prevent a simple calibration of the MDI magnetograms. For the considered time period, the evaluation of the MDI data shows an offset by about 8 Gauss (MAE) and no correlation between the data series (correlation-coefficient of −0.1). The application of ITI leads to an intercalibration of the LOS magnetograms that accounts both for the offset (MAE of 0.8 Gauss) and mitigates the differences in scaling (correlation-coefficient of 0.6), leading to a more consistent data series. For a direct comparison we track the active region NOAA 11106 from 2010-09-13 to 2010-09-19. In Fig. 4 we compare SOHO/MDI magnetograms and enhanced ITI magnetograms to the SDO/HMI magnetic field measurements. The ITI magnetograms show a clear increase in sharpness and better identification of magnetic elements as compared to the SOHO/MDI data. Differences to the reference magnetogram occur at the smallest scales, where the SOHO/MDI resolution limit is reached, while the overall flux distribution is in good agreement.

Fig. 3: Comparison of simultaneously observed SOHO/MDI, ITI enhanced, and SDO/HMI LOS magnetograms (B LOS).

a Visual comparison of the global calibration differences, showing observations from 2010-05-13 19:12 (top) and 2010-08-16 12:48 (bottom). The MDI magnetograms show global inhomogeneities, that prevent a simple calibration of the series. ITI largely mitigates these variations, leading to more consistent observations, that are in agreement with the HMI reference. b Unsigned magnetic flux density of the three magnetogram series for the period of parallel MDI and HMI observations starting from 2010-09-13. The MDI series shows substantial fluctuations, that are mitigated by ITI. The absolute error (MAE) and cross-correlation (CC) evaluations indicate a strong improvement in similarity between the ITI adjusted series and the HMI reference.

Fig. 4: Direct comparison of the active region NOAA 11106 over seven consecutive days.

The LOS magnetograms cover 150″ × 150″. ITI images show a similar appearance and sharpness to the SDO/HMI observations, while being also consistent with the low-resolution SOHO/MDI observations. The blue shaded area in Fig. 3b indicates the time frame of the images shown.

Image super resolution with different field-of-view—SDO/HMI-to-Hinode/BFI

Hinode/BFI provides a large data set of high-resolution partial-Sun continuum images at a wavelength band centered at 6684 Å, similar to the full-disk continuum images of SDO/HMI (centered at 6173 Å). The observations by Hinode/BFI cover various regions of the Sun which makes them suitable as high-quality target for the enhancement of HMI full-disk continuum observations.

Using unpaired image translation, we do not require a spatial or temporal overlap between the data sets, moreover the model training is performed with small patches of the full images. This enables the use of instruments that can observe only a fraction of the Sun for the enhancement of full-disk observations.

Here, we resize Hinode/BFI observations to 0.15 arcsec pixels and use ITI to super resolve HMI observations by a factor of 4. Hinode/BFI provides a spatial sampling of up to 0.0541 arcsec pixels, for our application we found that a resolution increase by a factor of 4 is already at the limit of where we can properly assess enhanced details.

In Fig. 5a we show an example of the full-disk HMI and ITI enhanced HMI observations. For a direct comparison, we manually extract two subframes from the HMI full-disk images and the Hinode/BFI image (panel b). The blue box shows the umbra and penumbra of the observed sunspot. The direct comparison demonstrates that the enhanced version is close to the actual observation and correctly separates fibrils that are only observed as blurred structure in the original HMI observation. The yellow box shows a pore and the surrounding solar granulation pattern. The granulation pattern is similar to the Hinode/BFI observations for larger granule, but shows differences in terms of shapes and inter granular lanes. Comparing these results to the original HMI observation, the deviations result mostly from diffraction limited regions that are not related to more extended solar features (e.g., extended fibrils, coherent granulation pattern). The correct generation of fibrils in the penumbra, that are beyond the resolution limit of HMI observations, show that the neural network correctly learned to infer information from the high-quality image distribution. The clear structures in the granulation pattern can be interpreted as a result of the perceptual optimization (training cycle ABA), but only partially agree with the reference observation.

Fig. 5: Evaluation of ITI translations of SDO/HMI continuum observations using simultaneous observations of SDO/HMI and Hinode/BFI continuum from 2013-11-18 17:46:20 and 2013-11-18 17:46:28, respectively.

a Original HMI (top left) and ITI enhanced (bottom right) full-disk observation. b Comparison of a 222″ × 222″ region and corresponding Hinode/BFI observation with full field-of-view and resolution. c Matched features of the original HMI, ITI enhanced and Hinode/BFI observation with 13″ × 13″ spatial extent. The penumbral features of ITI match the real observation. The small scale separation of the individual fibrils is beyond the resolution limit of HMI, but are largely reconstructed in the ITI observation. The coarse shape of the granulation pattern and the solar pore match the Hinode/BFI observation, while smaller intergranular lanes show deviations. d Example of the comparison between ITI and the deconvolution approach (2014-12-19 22:33). We show the original HMI image, the enhanced images and the absolute error maps to the original Hinode/BFI image. The ITI image shows a better agreement with the high-resolution reference, as can be seen from the reduced errors in the penumbra and sharper boundaries of the pores. An animation of the temporal evaluation from 2014-11-22 is provided in Supplementary Movies 1 and 2.

Both instruments show a substantial overlap that allows for a quantitative assessment of our method. Here, we compare our method to a state-of-the-art Richardson-Lucy deconvolution⁴⁷, using the point-spread-function of the HMI telescope from⁴⁸, and calibrating the pixel counts to the Hinode/BFI scale (see, “Methods” subsection “Data pre-processing”). In Fig. 5d we show an example for paired image patches. Both the deconvolved image and the ITI enhanced image show a general increase in perceptual quality. At smaller scales the ITI image shows a better agreement with the Hinode/BFI reference image. This can be best seen from the difference images, that show a smaller deviation of the penumbra and sharper boundaries of pores than for the deconvolved image. In contrast, the quiet-Sun region shows a slightly increased error for ITI, which can be interpreted as a mismatch of the enhanced granules.

Mitigation of atmospheric effects—KSO Hα low-to-high quality

The mitigation of atmospheric effects in ground-based solar observations imposes two major challenges for enhancement methods. (1) Since observations are obtained from a single instrument, there exists no high-quality reference for a given degraded observation. (2) The diversity of degrading effects is large, which commonly leads to reconstruction methods that can only account for a subset of degrading effects.

We use ITI to overcome these challenges by translating between the image domains of clear observations (high-quality) and observations that suffer from atmospheric degradations (low-quality) of the same instrument. We use ground-based full-disk Hα filtergrams from KSO that we automatically separate into high- and low-quality observations¹⁷. For all observations we apply a center-to-limb variation (CLV) correction and resize the observations to 1024 × 1024 pixels (see, “Methods” subsection “Data pre-processing”). In the resulting low-quality data set we include regular low-quality observations (e.g., clouds, seeing), while we exclude strong degradations (e.g., instrumental errors, strong cloud coverage).

In Fig. 6a we show two samples where a low-quality and high-quality observation were obtained within a 15 min time frame. We use these samples for a direct comparison of the ITI enhanced images (middle row Fig. 6a), which show a removal of clouds and are in agreement with the real high-quality observation.

Fig. 6: Evaluation of ITI translations for the mitigation of atmospheric degradations.

a Comparison of the ITI mitigation of atmospheric effects during varying observing conditions. From top to bottom we show low-quality KSO observations, the ITI reconstructed observations and reference high-quality observations that were taken minutes after the low-quality observation. We show two samples (800″ × 800″), where clouds are present in the low-quality KSO observation. The ITI reconstruction leads to clearer and unobstructed observations, where small chromospheric features remain unchanged and appear sharper. b Estimated image quality distribution of the original low-quality KSO observations (blue) and the ITI enhanced observations (orange). The red dashed line indicates the 0.25 quality threshold. c Three full-disk images with the lowest image quality after the ITI enhancement (about 0.24).

We use the manually selected low-quality observation from Jarolim et al.¹⁷ to provide a quantitative evaluation of the quality improvement. We adapt the image quality metric from ref. ¹⁷ for CLV corrected observations and only consider high-quality observations for model training, such that the quality metric estimates the deviation from the preprocessed synoptic KSO observations. In Fig. 6b we show the quality distribution of low-quality KSO observations and the ITI enhanced observations. The distributions show that ITI achieves a general quality increase, where the mean quality improves from 0.27 to 0.21, for the considered data set. In Fig. 6c we show three samples where the quality value intersect with the low-quality distribution after the ITI enhancement. The samples show that dense clouds and the contrail can be strongly reduced but not fully removed, which leads to the reduced image quality.

Approximation of observables—STEREO/EUVI-to-SDO/HMI magnetograms

In case of missing observables, a reasonable approach is to estimate the missing information based on proxy data. The STEREO mission observes the Sun in four EUV channels, but has no magnetograph onboard. We use ITI to complement the STEREO observations, by generating LOS magnetograms based on STEREO EUV filtergrams. Solar far-side magnetograms with deep learning were first obtained by Kim et al.⁴⁹, by training a neural network for SDO data and applying it to STEREO observations. In this study, we directly translate STEREO EUV filtergrams to the SDO domain. Similar to the application for inter-calibration of multi-instrument data, we use the EUV filtergrams of STEREO and SDO as low- and high-quality domain respectively. In addition, we add the SDO LOS magnetograms to the high-quality domain. Thus, the generator AB translates sets of images with four channels to sets of images with five channels. For the magnetograms we use the unsigned magnetic flux, in order to prevent unphysical assumptions on the magnetic field polarity, which can not be deduced solely from EUV images (see, Supplementary section Comparison to paired image-to-image translation).

For our training setup, we use a discriminator model for each channel and a single discriminator for the combined set of channels. This provides three optimization steps that enable the estimation of realistic magnetograms. (1) The single channel discriminator for the magnetograms assures that the generated magnetograms correspond to the domain of real magnetograms, independent of their content. (2) The content of the image is bound by the multi channel discriminator, that verifies that the generated magnetograms are consistent with EUV filtergrams of the SDO domain. (3) From the identity transformation B-B, we truncate the magnetogram from the SDO input data and enforce the reconstruction of the same magnetogram (see, methods subsection Model).

The different vantage points of STEREO and SDO do not allow for a direct comparison of estimated ITI and real SDO/HMI magnetograms, but partially overlapping observations were obtained by SOHO/MDI at the beginning of the STEREO mission (November 2006). Here, we use these SOHO/MDI magnetograms as reference to estimate the validity of the synthetic ITI magnetograms. Figure 7 shows two examples of STEREO/EUVI 304 Å observations, the corresponding ITI magnetograms and the real absolute SOHO/MDI reference magnetograms. We note a good agreement of the position of sunspots, which are not obvious from the EUV filtergrams. The magnetic elements appear similar in their distribution and magnitude, although we note a slight overestimation of magnetic field strength. The two examples show the variations of the generated magnetograms, where the magnetogram in panel a is similar to the actual observation, while the magnetogram in panel b shows more deviations. The biggest differences originate from the overestimation of magnetic field strengths, which especially affects regions with strong magnetic flux, where extended magnetic elements can be falsely contracted to a sunspot (i.e., spurious sunspot in panel b). An analysis of the temporal stability and the accompanying movie are given in the Supplementary Material (Supplementary Movie 3).

Fig. 7: Comparison between the synthetic ITI and overlapping SOHO/EIT magnetograms.

In panels a and b we show observations from 2006-12-27 13:20 and 2007-01-07 19:20, respectively. STEREO/EUVI observations of the 304 Å channel are given for comparison and scaled by their maximum and minimum value. The magnetograms show the absolute values of the LOS magnetic field strength and are scaled linearly to 2000 Gauss. The flux distribution at the full-disk scale is in agreement between the ITI and SOHO/MDI observations. The comparison of the active region in a shows that the ITI magnetogram matches the overall flux distribution. The active region in b shows larger deviations that mainly originate from confined regions (i.e., sunspots) where ITI overestimates the magnetic field strength. An animated version is provided in the Supplementary Materials (Supplementary Movie 3).

A direct application of the far-side magnetograms are full-disk heliographic synchronic maps of magnetic flux. We combine the ITI magnetograms from STEREO A and B together with the real SDO/HMI magnetograms into heliographic maps. On the left in Fig. 8 we show the magnetograms over 13 days, where the active regions (blue circle) are centered for STEREO B, SDO and STEREO A, respectively. On the right in Fig. 8, we show the EUV filtergrams for comparison (c.f., see results subsection Inter-calibration of multi-instrument data). The overall distribution of magnetic flux and the position of the major sunspots is consistent with the SDO observation, although we note that again small sunspots appear that are inconsistent with the SDO/HMI observation (panel a). Footprints of other magnetic features, such as the filament (green circle), can also be identified in the ITI magnetograms. Where a similar flux distribution can be seen in all three magnetograms.

Fig. 8: Qualitative evaluation of synchronic heligraphic maps obtained from SDO/AIA+HMI and ITI enhanced STEREO/EUVI data.

We show three consecutive observations of heliographic synchronic magnetograms displaying the absolute value of the LOS magnetic field strength (left) and reference EUV maps in three different filters (304, 193, 211 Å; right). We compare ITI magnetograms obtained from STEREO EUV filtergrams with real observations from SDO/HMI. All magnetograms are scaled to 1000 Gauss. We show the same active regions (blue circle) as observed by each instrument over 16 days. In the top row and bottom row the active regions are observed by STEREO B and STEREO A, respectively. The comparison to the real SDO/HMI observation (middle row) shows that ITI detects a similar magnetic flux distribution. Both ITI magnetograms show sunspot configurations with the characteristic tilt of preceding and following sunspot (Joy’s law), but deviate from the SDO/HMI observation in terms of stronger and more confined magnetic fields (i.e., more sunspots). The green circle indicates the magnetic field configuration of the filament, that can be seen in the EUV observations. The ITI magnetograms show a similar magnetic flux distribution to the SDO/HMI observation. An animated version of this figure is provided online (Supplementary Movie 4). ITI performs a domain translation that also includes the EUV channels. As can be seen from the heliographic EUV maps, the translated EUV channels smoothly integrate with the SDO observations.

Quantitative evaluation

In this section we provide a quantitative evaluation of our results from the previous applications. For all data set pairs we evaluate the perceptual improvement by estimating the distance between the individual image distributions, as evaluated by the Fréchet inception distance (FID). The FID is a commonly used metric for GANs to estimate the quality of synthesized images and is a measure for the distance between two image distributions⁵⁰. We use this metric to quantify the similarity between the low-quality and ITI reconstructed images to the high-quality reference distribution. As a baseline, we compute the distance between the low-quality dataset to the high-quality dataset. We consider each image channel separately and compute the FID over the full test set. The results in Table 2 show that throughout ITI enhanced images are closer to the high-quality image distribution than the low-quality distribution. This indicates that our method is able to map images closer to the high-quality distribution, or in other words, leads overall to a perceptually higher image quality.

Table 2 Quantitative evaluation of ITI translations and baseline calibrations

We utilize paired data samples from our test sets to provide a quantitative evaluation of our ITI method. To estimate the image quality we evaluate the peak-signal to noise ratio (PSNR), the structural similarity index (SSIM) and the image cross correlation (CC). For the pixel-wise comparison we co-register the corresponding image pairs based on the phase-correlation of the Fourier-transformed images⁵¹. Note that while this allows for a better alignment, this approach can not achieve a perfect pixel-wise alignment. Among the considered applications we find a sufficient amount of paired samples between observations from SOHO/EIT + MDI and SDO/AIA+HMI, as well as SDO/HMI and Hinode/BFI. For the KSO observations we obtain a small set of 16 paired samples, while there is no overlap between observations from SDO/AIA and STEREO/EUVI.

For the SOHO/EIT + MDI observations we use 121 simultaneous full-disk observations from 2010-05-13 to 2011-04-11, and compare them to the corresponding co-registered SDO/AIA+HMI observations. As baseline we consider the baseline calibration of the mean and standard deviation between the data sets (see, methods subsection Calibration). For all EUV images we apply the asinh stretching described in methods subsection Data pre-processing, to ensure an equal contribution of more faint regions (e.g., quiet-Sun regions). Overall, the pixel-wise metrics show limited improvement. Significant improvements in terms of PSNR are achieved for the 211 and 304 Å channels, which we attribute to the ITI intercalibration. For the 211 Å filter throughout an improved image quality is achieved, which can be related to differences between the 284 and 211 Å channel which are mitigated by the ITI translation. In contrast, the 304 Å channel shows a decreased SSIM and CC, which suggests that the synthesized small scale structures do not match the reference image.

For the translation of SDO/HMI continuum observations to Hinode/BFI data, we find 57 observations that can be appropriately aligned (<10s difference in the observation time). As baseline we use the calibrated HMI images deconvolved with the Richardson-Lucy method. We obtain throughout an improvement in PSNR, SSIM and CC, this can be interpreted as an improved agreement of super-resolved features with the reference observations. Note that for the baseline we computed the calibration from the full solar disk (i.e., mean and standard deviation over all pixels on the solar disk). Computing the baseline calibration from the co-aligned images would improve the PSNR. With our ITI method we can perform a more consistent calibration of the full solar images.

To quantify the ability to mitigate atmospheric degradations, we select Hα observations from KSO where high-quality observations are obtained within 15 min after a low-quality observation. We identify 16 observations across our test set where this condition is satisfied. For each of the original pre-processed observations and ITI translated images we perform a differential de-rotation to the reference observation, select the central 512 ×  512 pixels, and co-align the image pairs based on the phase-correlation. With this we aim to spatially co-register the observations independent of the off-limb region. Both before and after the ITI translation, the SSIM and CC are low. This can be explained by the temporal changes in the solar atmosphere which limit the evaluation of structural similarities. The PSNR indicates that differences due to larger inhomogeneities across the solar disk (e.g., clouds) are reduced by ITI.

Note that for the pairing we typically obtain the best matching for the continuum observations from SDO/HMI and Hinode/BFI, due to the small field of view and the more prominent features present in the images (e.g., dark sunspots). The alignment of full-disk observations is largely dominated by matching the solar limb and adjusting for the rotation angle. While we can partially account for the solar rotation of the KSO observations, the temporal evolution of the solar atmosphere leads to notable changes. The results are summarized in Table 2.

Discussion

In this study, we present a framework for the homogenization of data sets from different instruments, enhancement of solar images and mitigation of quality degradations. We apply unpaired image-to-image translation to use high-quality observations to enhance and intercalibrate observations of lower quality. Our framework does not require any spatial or temporal overlap and can account for the translation between instruments with different field-of-view, resolution and calibration, which makes our method applicable to many astrophysical imaging data sets. The approach is purely data driven, which avoids assumptions about the mapping between high- and low-quality samples and improves the applicability to actual observations.

The use of paired domain translations between different instruments typically requires extensive pre-processing, where observations need to be matched in time and co-aligned at the pixel-level¹⁶. Both operations can typically not be perfectly satisfied and strongly reduce the available data for model training. Here, unpaired domain translation enables an additional range of applications, without extensive pre-adjustments, and comparable performance to paired translations (see, Supplementary section Comparison to paired image-to-image translation and ref. ²¹).

The four applications were selected to provide a comprehensive evaluation of the developed ITI method. We obtain throughout an assimilation of the target image distribution, as evaluated by the FID and in agreement with our optimization target. Specifically, the quality metric for ground-based observations indicates a significant correction of atmospheric degradations (Quality improvement from 0.27 to 0.21; see, “Results” subsection “Mitigation of atmospheric effects”). The time series of average EUV intensity and unsigned magnetic flux indicate an inter-calibration that exceeds baseline calibration methods (see, “Results” subsection “Inter-calibration of multi-instrument data”). Note that this inter-calibration is a direct result of the translation of individual images and does not explicitly depend on the computation of data set statistics (e.g., adjustment of mean and standard deviation).

We evaluated paired low- and high-quality observations, which are more difficult to interpret due to the sparsity of simultaneous observations, the absence of temporally aligned observations, and the problematics of pixel-wise image pair co-registration. The evaluation suggests that our method provides significant improvements when accounting for different filter channels (e.g., SOHO/EIT 284 Å to SDO/AIA 211 Å). Here, the multi-channel information provides a primary advantage to mitigate differences in observables. We showed that with the simultaneous translation of image channels, and requiring the consistency of the combined set of channels, information on the image content can be exchanged between related channels (SOHO-to-SDO; Fig. 2b). Overall, the pixel-wise metrics show small or no improvements, which is partially induced by small shifts between the co-aligned image pairs, temporal differences between the paried images, or the noise level. For the KSO observations the 15 min time difference causes already significant differences in the structure of the Hα observations (SSIM of 0.46). On the other hand, the comparison of light-curves and total unsigned magnetic flux (see, results subsection Inter-calibration of multi-instrument data), indicate an improvement over the global data set (e.g., improvement of MAE by about 50% for EUV light-curves, and MAE improvement from 7.89 to 0.75 Gauss for magnetic flux density). Therefore, our method provides an overall homogenization between data sets, while features at the resolution limit should not be considered for further analysis.

The comparison of SDO/HMI super-resolved observations with reference Hinode/BFI observations suggests that information can be drawn from the high-quality image distribution to enhance spatially extended structures. On the other hand, enhanced features at the resolution limit of the telescope are typically not reliable. This owes to the fact that the spatial information is degraded beyond the point of reconstruction, and the enhanced features solely match a common pattern of the high-quality distribution (e.g., small granules). Specifically when dealing with larger differences in image quality (e.g., SOHO/EIT 304 Å to SDO/AIA 304 Å), small scale structures are not in agreement with the reference observations, as can be seen from the reduced SSIM score.

Synthesizing additional observables in the form of unsigned magnetic flux from EUV information is the most challenging application. Our method shows that the estimated flux distribution of quiet-Sun regions, active regions, and solar filaments is similar to reference magnetograms, but individual magnetic elements typically only achieve a limited agreement. While this can be a reasonable approximation to estimate potential space weather hazards of active regions⁴⁹, a more physics-based approach is required to deduce reliable magnetic field information from EUV observations.

A notable advantage of our ITI method is the applicability to full-resolution observations (e.g., 4k × 4k pixels), where it provides both small scale enhancements as well as global corrections (e.g., calibration, mitigation of atmospheric degradations).

Enhanced images can be used for a better visualization of blurred features or to mitigate degradations, but they should not be used for studying small scale features. The main application lies in the data calibration and assimilation, which allow us to study homogenized data sets of multiple instruments and to apply automated algorithms to larger data sets without requiring additional pre-processing steps⁴⁶. Similarly, algorithms that perform feature detections or segmentations can profit from enhanced images (e.g., identification of magnetic elements, granulation tracking, filament detection).

The demonstrated applications are of direct use in solar physics. (1) The homogenized series of space-based EUV observations provides a means for the joint use of the three satellite missions. The data set provides a resource for the study of solar cycle variations (cf.^2,45), contributes additional samples for data-driven methods and enables the application of automated methods that were developed specifically for SDO/AIA data to the full EUV data series^46,52,53. (2) The results of the HMI-to-Hinode translation allow for a continuous monitoring of the Sun in a resolution of 0.15 arcsec pixels, producing full-disk photospheric images with increased resolution. As can be seen from Fig. 5, the high-resolution observations of Hinode/BFI provide only a partial view on the objects of interest. The enhanced HMI images can provide useful context information by accompanying high-resolution observations. This applies to both the spatial extent as well as the extent of the time series by additional observations before and after the Hinode/BFI series. (3) The correction of atmospheric effects in ground-based observations can be operated in real-time (approximately 0.44 seconds per observation on a single GPU). This allows us to obtain a more consistent observation series and assists methods that are sensitive to image degradations (e.g., flare detection;⁵⁴). (4) The generated STEREO far-side magnetograms give an estimate of the total magnetic flux distribution, which can provide a valuable input for space-weather monitoring⁴⁹. Information about the magnetic polarities is required for the further application to global magnetic field extrapolations²⁶. From the patch-wise translation the inference of global magnetic field configurations (e.g., Hales law), would be arbitrary and was therefore omitted in this study. For all the considered applications, our trained model processes images at higher rates than the cadence of the instruments, allowing the application in real-time and fast reconstruction of large data sets.

With these applications we provide an approach for solar image enhancement, paving the way for advanced domain translations that require no spatial or temporal overlap. The results could be further improved by considering, for example, additional temporal information (e.g., video sequences), or by incorporating additional physical constrains in the image translation⁵⁵. A general shortcoming of image super-resolution is the inherently ill-posed problem, which could be reduced by incorporating additional information. This could be achieved, for example by building on image bursts, and the ability of our method to draw information from the multi-channel representation.

The extension to other data sets requires the acquisition of a few thousand images of low- and high-quality, where an alignment is not required, and the training of a new model. The translation is performed for observations of the same type (e.g., LOS magnetogram) or in the same (or similar) wavelength band with similar temperature response (e.g., EUV), but with different image quality, either reduced by atmospheric conditions or by instrumental characteristics, such as spatial resolution. The data sets need to provide similar observations in terms of features and regions, in order to avoid translation biases⁵⁶. The estimation of magnetic field information based on EUV filtergrams illustrates an example where this condition is not strictly required. Here the image translation is constrained by the multi-channel context information and the learned high-quality image distribution.

In conclusion, our study demonstrates the effectiveness of unpaired image-to-image translation for solar physics and provides a promising step to address common problems of astrophysical image processing. Commonly, paired samples are not available between different instruments or are difficult to obtain, underscoring the importance of domain translation approaches. By leveraging high-quality data sets, our fully data-driven approach can bridge the gap between instrumental characteristics and provides an informed image enhancement.

Methods

This section provides an overview of the instruments and data sets used for the individual applications as well as the performed pre-processing steps. In subsection “Model” we describe the model architecture and training setup. Subsection “Evaluation” summarizes the metrics used for the evaluation.

Datasets

Instruments

Here, we describe the characteristics of the considered instruments for the evaluation of our method. The pair-wise correspondence of the data sets is discussed in the Results section.

The solar dynamics observatory (SDO) is a space-based mission located in a circular geosynchronous orbit, that provides science data since 1 May 2010³⁷. For our study we use EUV filtergrams from the atmospheric imaging assembly (AIA³⁶), and line-of-sight (LOS) magnetograms and continuum observations from the helioseismic and magnetic imager (HMI³⁸). Both instruments provide solar full-disk images with a spatial sampling of 0.6 arcsec pixels and a spatial resolution of 1.5 arcsec. The data are recorded by 4096 × 4096 pixel CCDs. The AIA instrument operates at a cadence of 12 s and provides EUV filtgrams in seven band passes. In this study, we consider Fe IX (171 Å), Fe XII, XXIV (193 Å), Fe XIV (211 Å) and He II (304 Å) filtergrams. The emission lines are primarily formed in the solar corona and chromosphere and are associated with peak temperatures of ion formation at 6.3 × 10⁵ K (Fe IX; quiet corona, upper transition region), 1.6 × 10⁶ K, 2.0 × 10⁷ K (Fe XII, XXIV; corona, hot flare plasma), 2.0 × 10⁶ K (Fe XIV; active-region corona) and 5.0 × 10⁴ K (He II; chromosphere, transition region)³⁶. The HMI instrument provides maps of the photospheric magnetic field strength and orientation by observing polarizations of the Fe I absorption line (6173 Å). The continuum observations are calculated from six points in the Fe I line³⁸. For both the HMI magnetograms and continuum observations, we use the 720 s series.

The solar and heliospheric observatory (SOHO) is a space-based mission, located at Lagrange point L1, that was launched in December 1995³⁵. In this study we use data from the extreme-ultraviolet imaging telescope (EIT³⁴) and LOS magnetograms from the Michelson doppler imager (MDI³⁹). Both instruments provide full-disk 1024 × 1024 pixels images of the Sun at a spatial sampling of 2.6 arcsec pixels and about 5 arcsec spatial resolution. From EIT we use Fe IX (171 Å), Fe XII (195 Å), Fe XV (284 Å) and He II (304 Å) filtergrams. The peak temperatures for ion formation are 1.3 × 10⁶ K, 1.6 × 10⁶ K, 2.0 × 10⁶ K and 8.0 × 10⁴ K, respectively³⁴. The MDI instrument is the predecessor of SDO/HMI and derives the LOS magnetograms in the Ni I 6768 A absorption line³⁹.

The solar terrestrial relations observatory (STEREO) is a twin-satellite mission that operates two identical satellites on two orbits close to 1 AU. The orbits are selected to lead to a yearly separation of the spacecrafts by about 45 degree, as viewed from the Sun³³. The mission was launched in 2006 and provides stereoscopic observations of the Sun since then. In this study, we use the exteme ulraviolet imager (EUVI³²) of the Sun-Earth connection coronal and heliospheric investigation (SECCHI⁵⁷) instrument. The imager provides full-disk filtergrams with 2048 × 2048 pixels with a spatial sampling of about 1.6 arcsec pixels and 3.2 arcsec spatial resolution⁵⁷. Similarly to SOHO/EIT, filtergrams of Fe IX (171 Å), Fe XII (195 Å), Fe XV (284 Å) and He II (304 Å) are recorded. The associated peak temperatures are in the same range as for the SOHO/EIT filters (see above).

The solar optical telescope (SOT) onboard the Hinode satellite (launched in 2006) is a 50 cm aperture telescope that provides high-resolution images of partial fields of the Sun⁴⁰. The instrument comprises multiple broad- and narrow-band filters and provides spatial resolutions of up to 0.2 arcsec and a pixel scale of 0.054 arcsec pixels. In this study, we use continuum observations centered at a wavelength of 6684 Å of the Broadband Filter Imager (BFI). The BFI instrument provides a field-of-view of 218" × 109", producing images with 4096 × 2048 pixels, and operates at a cadence  <10s⁴⁰. Observations are recorded on demand.

Kanzelhöhe Observatory for Solar and Environmental Research (KSO; https://www.kso.ac.at/) provides ground-based solar full-disk Hα filtergrams at a spatial resolution of 2 arcsec. The data are recorded by a 2048 × 2048 pixel CCD corresponding to a sampling of about 1 arcsec pixels. KSO regularly takes Hα images at a cadence of 6 s and provides a fully automated data reduction and data provision pipe line, which allows for data access in near real time^41,42,58. The current instrument setup is in operation since 2008. The Hα line is formed by absorption at 6563 Å in the solar chromosphere and by cooler plasma in the solar corona (filaments).

Datasets

Supplementary Table 1 summarizes the considered instruments, observing wavelengths, the amount of data samples and the number of independent patches that can be extracted. The images are normalized such that saturations are avoided, which we found beneficial for model training. We apply a strict temporal separation of our train and test set, where we use a buffer of one month between training and test set to avoid a potential memorization⁵⁹. In general, we always consider February to September for our training set, while observations from November and December correspond to the test set. In addition, we exclude the SDO/AIA+HMI observations from 2010 that overlap with the SOHO/EIT + MDI observations from the training set.

The observations from SDO, STEREO and SOHO are taken at high cadence. For each data set we randomly sample observations from the full mission lifetime (SDO: 2010-2020, http://jsoc.stanford.edu; STEREO: 2006-2020, Solar Data Analysis Center via http://virtualsolar.org; SOHO: 1996-2011, STEREO Science Center via http://virtualsolar.org). For Hinode/BFI observations we use 2 × 2 binned red continuum observations at 6684 Å of the SOT Broadband Filter Imager (BFI) as high-quality reference. In the time between 2007 and 2016 we select a single observation from each observation series taken with BFI (https://darts.isas.jaxa.jp/solar/hinode). For the KSO Hα, we utilize the image quality assessment method by Jarolim et al.¹⁷, to automatically assemble a data set of ground-based KSO observations that suffer from atmospheric degradations (e.g., clouds, seeing), in the time between 2008 and 2019. As high-quality reference we use the synoptic archive of contrast enhanced KSO observations, that comprises manually selected observations of each observing day. In addition we perform a quality check and remove all observations of reduced image quality (https://www.cesar.kso.ac.at).

With the use of unpaired data sets there is typically no limitation in data samples. For all our applications we select a few thousand images and extend them in case of diverging adversarial training.

Data pre-processing

For each observation we center the Sun, normalize the field-of-view to 1.1 solar radii and crop the frame, such that the extent of the solar-disk is independent of yearly variations caused by the elliptic orbit (cf.⁶⁰). We correct for instrumental degradation and normalize the exposure time⁴⁴. For STEREO/EUVI, SOHO/EIT, Hinode/BFI we use eit_prep, secchi_prep and fg_prep routines, respectively, provided by the IDL package SolarSoft. For SDO/AIA we use the SDO autocalibration by¹⁸. The STEREO/EUVI instrument shows significant degradations of the 304 Å channel, that are not considered by the STEREO prep routine (about 50%). We correct for this degradation by extracting the quiet-Sun regions over the full data set and fitting the degradation as a first-order polynomial. The maximum intensity value of each instrument channel is estimated from the maximum intensity values over the full data set by ${\hat{I}}_{max}=\,{\mbox{mean}}({I}_{max})+0.5\cdot {\mbox{std}}\,({I}_{max})$, where I_max refers to the maximum intensity values of the images and std to the standard deviation. We clip negative values and values larger than the estimated maximum intensity and normalize to [0, 1]. We apply an arcsinh stretch for EUV filtergrams

$$\hat{x}=\frac{\,{{\mbox{asinh}}}(x/a)}{{{\mbox{asinh}}}\,(1/a)}2-1,$$

(1)

where x refers to the input, $\hat{x}$ to the scaled output and a is set to 0.005. This stretching function provides for a logarithmic behavior for large values and a linear behavior for small values. We scale the data afterwards to the interval [−1, 1], such that the data is suitable for a $\tanh$ activation function. For LOS magnetograms we use a linear normalization of values between [−3000, 3000] Gauss to [−1, 1] and set all off-limb values to zero.

For the continuum observations of SDO/HMI and Hinode/BFI we analogously estimate the maximum value and scale linearly between 0 and the maximum value.

For all KSO observations we shift the Sun to the image center, crop the field-of-view to 1 solar radius and resize it to 1024 × 1024 pixels for the low-to-high translation. The center-to-limb variation is corrected by plotting the theoretical correction $\mu=\cos \theta$ against the intensity values of the image, where θ refers to the heliocentric angle, and fitting a fourth order polynomial that gives the intensity correction I_corr(μ) (cf.⁶¹). We scale values between 0.65 and 1.5 to [−1, 1] with an arcsinh stretch (a = 0.5). We apply the preprocessing per frame and set all off-limb pixels to  −1.

Model

Generative adversarial networks (GANs) have shown the ability to generate highly realistic synthetic images^62,63,64. The training is performed in a competitive setup of two neural networks, the generator and discriminator. Given a set of images, the generator maps a random input from a prior distribution (latent space) into the considered image domain, while the discriminator is trained to distinguish between real images and synthetic images of the generator. The generator is trained to compete against the discriminator by producing images that are classified by the discriminator as real images. The iterative step-wise optimization of both networks allows the generator to synthesize realistic images and the discriminator to identify deviations from the real image distribution⁶². By replacing the prior distribution with an input image, a conditional mapping between image domains can be achieved^65,66.

We use a GAN to generate highly realistic low-quality observations that show a large variation of degrading effects. We propose an informed image enhancement which uses domain specific knowledge to infer missing information. For this task, we employ a GAN to model the high-quality image distribution and constrain enhanced images to correspond to the same domain. With the use of a sufficiently large data set we expect that we can learn to correctly model the true image distributions and find a mapping that is applicable for real observations.

The primary aim of our method is to transform images from a given low-quality domain to a target high-quality domain. We refer to the high-quality domain as B and to the low-quality domain as A. In order to achieve an image enhancement that can account for various image degradations we aim at synthesizing realistic images of domain A based on images of domain B. The pairs of high-quality and synthetic low-quality images are used to learn an image enhancement. Thus, the training process involves mappings from A to B (A-B), as well as mappings from B to A (B-A).

The model setup involves four neural networks, two generators and two discriminators. The generators learn a mapping between A-B and B-A. The discriminators are used to distinguish between synthetic and real images of domain A and B. The training cycle for image enhancement uses a high-quality image (B) as input, which is translated by the generator BA to domain A. The synthetic degraded image is then restored by the generator AB (Fig. 1). We optimize the generators to minimize the distance between the original and reconstructed image, as estimated by the reconstruction loss (cycle consistency). The simplest solution for this setting would be an identity mapping by both generators. We counteract this behavior with the use of discriminator A, which is trained to distinguish real images of domain A from synthetic images of generator BA. With this we constrain the generator BA to generate images that correspond to domain A and the generator AB to restore the original image from the degraded version. With the synthesis of more realistic low-quality images, we expect the generator AB to perform equally well for real low-quality images.

Similarly to Wang et al.⁶⁵ each discriminator network is composed of three individual networks, where we decrease the image resolution by a factor 1, 2, 4 for the three networks, respectively. With this the perceptual quality optimization is performed at multiple resolution levels, thus estimating small-scale features as well as more global relations.

We follow the training setup of Zhu et al.²¹ and use three additional training cycles. The second cycle enhances the perceptual quality of high-quality images with the use of discriminator B. The mapping is again performed under the cycle consistency criteria. Thus, we start with a low-quality image and perform an A-B mapping, followed by a B-A mapping. The additional training with discriminator B ensures that images produced by generator AB correspond to domain B, which adds an additional constrain for image enhancement and improves the perceptual quality. The last two training cycles ensure a consistency under identity transformations. To this aim, we translate images of domain A with the generator BA and images of domain B with the generator AB, and then minimize the reconstruction loss between the original and transformed images. For differences in resolution we use bilinear upsampling and average pooling for downsampling.

By only using an image as input to our generator, the results are deterministic⁶⁷. In the present case we are interested in modeling various degrading effects and explicitly want to avoid the generation of synthetic noise, based on image features (e.g., solar features, instrumental characteristics). This task is often addressed as multimodal translation^22,23 and also relates to style transfer between images^64,68,69. Here, we add an additional noise term to our generator BA, so that multiple degraded versions can be generated from a single high-quality image. For the generator AB we assume that there exists a unique high-quality solution for a given low-quality observation.

The cycle consistency of low-quality images is ensured by first estimating the noise of the original low-quality image. For this task, we employ an additional neural network which we term noise-estimator. We use the noise-estimator for the A-B-A and A-A mapping and randomly sample a noise term for the B-A-B mapping from a uniform distribution [0, 1]. The advantage of this approach is two fold. (1) The mappings A-B-A and A-A are not ambiguous, which allows for a clear separation of noise from high-quality images. (2) The explicit encoding of low-quality features into the noise term representation benefits the relation between the generated low-quality features and the noise term, by enforcing the use of the noise term in the generator⁶⁹. For both the A-B-A and A-A mapping we minimize the distance between the estimated noise of the original image and the estimated noise of the reconstructed image (see, methods subsection Losses). This approach relates to image style transfer (e.g.,^64,69,70), where we consider the low-quality features as style that we transfer to the high-quality images.

Image quality can be addressed in terms of perceptual (e.g., mean-squared-error) and distortion (e.g., Wasserstein distance) quality metrics. The spanned space by both metrics is referred to as perception-distortion-plane, where the minimum on both metrics is not attainable⁷¹. In order to obtain the best quality, image enhancement algorithms should consider both metrics for optimization. This can be seen from image translation tasks, where the additional use of an adversarial loss achieved significant improvements in terms of image quality^65,66. We employ content loss, which shows a better correspondence to image features than pixel based metrics (e.g. MSE), as primary distortion metric and use adversarial loss of GANs for the perceptual optimization. From this setup we aim at achieving an optimal minimum in the perception-distortion plane.

Most full-disk observations exceed the computational limit when used in their full resolution. This can be overcome by reducing the resolution of the image or by training the neural network with image patches. In this study, we use image patches, in order to provide images with the highest resolution attainable. After model training, the neural network can be applied to full-resolution images, which is in most cases computationally feasible.

As described in Cohen et al.⁵⁶, the use of data sets with an unequal distribution of features is likely to produce artifacts. For this reason we balance during training between image patches with solar features (e.g., active regions, filaments) and quiet Sun regions. During training we randomly sample patches from a biased distribution. For Hinode/BFI we additionally consider the solar limb, such that patches are equally sampled across the full solar disk.

Multi-channel architecture

In the case of multiple image channels (e.g., simultaneously recorded filtergrams at different wavelengths) we employ multiple discriminator networks. We use an individual discriminator for each of the image channels (single-channel) and an additional discriminator that takes the combined channels as input (multi-channel). Here, each discriminator represents again a set of three single discriminators at three different resolution scales. From the usage of the single-channel discriminators we expect a better matching of the individual channel domains without influences of the other channels. The multi-channel discriminator is capable to assess the validity between the channels (e.g., appearance of features across the channels). This concept is especially important for the estimation of observables (see, results subsection Approximation of observables), where the single-channel discriminator solely addresses the perceptual quality of the estimated observable, while the multi-channel discriminator restricts the estimated observable to be valid within the context of the other channels.

We note that the multi-channel architecture has no strict mapping requirements (e.g., same filters for each channel), such that the number of input and output channels is flexible. Thus, additional observables can be approximated (e.g., more output channels than input channels) and a conversion between similar channels can be achieved (e.g., SOHO/EIT 195 Å to SDO/AIA 193 Å).

Model architecture

For image translation tasks the concept of multi-scale architectures has shown great success. Skip connections have demonstrated the ability to enhance training performance and to preserve spatial details^72,73.

For our generators we employ convolution blocks similar to Karras et al.⁷⁴ and introduce skip connections at each resolution level, similar to Ronneberger et al.⁷³. The full overview of our generator models is given in Supplementary Fig. 1. We use convolutional blocks composed of a convolutional layer with a kernel size of 3 × 3, followed by an instance normalization⁷⁵ and a ReLU activation. In order to retain the image dimensions and to avoid boundary artifacts, we use reflection padding before the convolution layers⁶⁵. Our network is composed of three downsampling blocks, an intermediate block, followed by three upsampling blocks. A downsampling block consists of n convolutional blocks followed by a convolutional block with stride 2 (downsampling by 2). While downsampling we double the number of filter channels. For an upsampling block we first apply bilinear upsampling, reduce the number of filter channels by a factor of 2 with a convolutional block and apply n additional convolutional blocks. At each resolution level we use skip connections between the downsampling and upsampling blocks. Hereby the features prior to downsampling are concatenated with the features after the first convolutional block in the upsampling block. The input image is transformed by an initial convolutional block with a kernel size of 7 × 7 and 64 filter channels. The three downsampling blocks are consecutively applied with n = 1, 2, 3. The intermediate layer consists of three convolution blocks with the same number of filters as the last downsampling block (512). The upsampling blocks are applied in the inverse order (n = 3, 2, 1). The output image is obtained by a convolution layer with kernel size 7 × 7, followed by a $\tanh$ activation function.

For the generator BA we include the noise term by concatenation with the features of the last downsampling block (prior to the intermediate block). The noise term is transformed by a convolutional block with 512 filter channels and matched to the dimensions of the last downsampling block by an upsampling block with n = 3 and 512 filter channels (dashed line in Supplementary Fig. 1).

Our networks are fully convolutional, therefore the training can be performed with image patches, while the evaluation is performed with full resolution images. For the noise term we use 16 channels and the spatial dimensions are adjusted to the considered image size (1/16 of the input image resolution). The instance normalization accounts for the global style transfer, while the noise term accounts for localized degradations in the image.

For the discriminators we use the architecture introduced in Wang et al.⁶⁵, where each discriminator is composed of three individual networks that operate on different scales. The discriminators are composed of four consecutive stride 2 convolutions with instance normalization and ReLU activation. We start with 64 filter channels and consecutively increase them by a factor 2 for each layer. The discriminators output is obtained by a convolution layer with one filter channel. Therefore, each discriminator provides a grid of predictions instead of a single output⁶⁶. For the noise estimator we use the same architecture as for the discriminators, adjust the number of filters of the last layer to the noise dimensions (16) and apply a final sigmoid activation function.

For an image enhancement that involves a resolution increase, we extend the generator AB by additional upsampling blocks and the generator BA by additional downsampling blocks. In the case of images with multiple channels (e.g., multiple filtergrams, magnetograms), we translate all channels simultaneously. For each image channel we use an additional discriminator that ensures the correct representation of the generated channel, independent of the other image channels. The primary discriminator is adjusted to take all filter channels into account simultaneously to ensure the consistency between the channels. For an unequal number of channels between domain A and B, we adjust the identity cycles by truncating additional channels and extending missing channels with zeros.

Losses

The cycle consistency serves as distortion metric for image enhancement. We use the mean absolute error (MAE; ${{{{\mathcal{L}}}}}_{{{{\rm{MAE}}}},{{{\rm{BAB}}}}}$ to compute the difference between the original high-quality image x_B and the model reconstruction ${G}_{AB}\left({G}_{BA}({x}_{B},z)\right.$

$${{{{\mathcal{L}}}}}_{{{{\rm{MAE}}}},{{{\rm{BAB}}}}}={\mathbb{E}}[{\parallel {x}_{{{{\rm{B}}}}}-{G}_{{{{\rm{AB}}}}}({G}_{{{{\rm{BA}}}}}({x}_{{{{\rm{B}}}}},z))\parallel }_{1}].$$

(2)

Here, G_AB and G_BA refer to generator AB and generator BA, respectively, and z refers to the added noise term. Pixel-based metrics (e.g., MAE) can prevent large divergences between the original image and the reconstruction but small shifts can cause a large increase of the reconstruction loss, which might provide a poor estimate of the image quality⁷¹. For this reason, we utilize in addition content loss, that compares feature similarity over pixel-wise similarity. Layer-wise activation of a neural network resemble extracted features and are therefore related to the content of the image. The content loss metric is computed by taking the mean-absolute-error (MAE) between the feature activation of the generated and original image. We define the content loss ${{{{\mathcal{L}}}}}_{{\mbox{Con,BAB}},j}$ based on the discriminator, similar to Wang et al.⁶⁵ as

$$\begin{array}{l}{{{\mathcal{L}}}}_{{{\mbox{Con,BAB}}},j}=\\ {\mathbb{E}}{\sum }_{i=1}^{4}\frac{1}{{N}_{i}}\parallel {D}_{{{\rm{B}}},j}^{(i)}({x}_{{{\rm{B}}}})-{D}_{{{\rm{B}}},j}^{(i)}({G}_{{{\rm{AB}}}}({G}_{{{\rm{BA}}}}({x}_{{{\rm{B}}}},z))){\parallel }_{1},\end{array}$$

(3)

where ${D}_{{{{\rm{B}}}},j}^{(i)}$ to the activation layer i of network j from discriminator B and N_i to the total number of features of activation layer i. For each of our discriminators we use all intermediate activation layers (cf.¹⁷).

For consistency of the estimated noise terms, we minimize the MAE between the estimated noise term of the original low-quality image and its reconstructed version (A-B-A)

$$\begin{array}{l}{{{{\mathcal{L}}}}}_{{{{\rm{Noise}}}},{{{\rm{ABA}}}}}=\\ {\mathbb{E}}[\parallel NE({x}_{{{{\rm{A}}}}})-NE({G}_{{{{\rm{BA}}}}}({G}_{{{{\rm{AB}}}}}({x}_{{{{\rm{A}}}}}),NE({x}_{{{{\rm{A}}}}}))){\parallel }_{1}],\end{array}$$

(4)

and similarly for the identity mapping (A-A)

$${{{{\mathcal{L}}}}}_{{{{\rm{Noise}}}},{{{\rm{AA}}}}}={\mathbb{E}}[\parallel {NE}({x}_{{{{\rm{A}}}}})-NE({G}_{{{{\rm{BA}}}}}({x}_{{{{\rm{A}}}}},{NE}({x}_{{{{\rm{A}}}}}))){\parallel }_{1}],$$

(5)

where NE refers to the noise-estimator, and x_A to the low quality image. To avoid the collapse of the noise term to a constant value, we minimize the MAE between the randomly sampled noise term and the estimated noise of the synthetic low-quality image (B-A-B)

$${{{{\mathcal{L}}}}}_{{{\rm{Noise}}},{{\rm{BAB}}}}={\mathbb{E}}[\parallel z-{NE}({G}_{{\rm{BA}}}({x}_{{\rm{B}}},z)){\parallel }_{1}].$$

(6)

Equations (2) and (3) refer to the B-A-B cycle. The other training cycles are computed analogously, where we use the discriminator of the corresponding domain for the extraction of activation features (i.e., D_A for images of domain A). For the total reconstruction loss ${{{{\mathcal{L}}}}}_{{{{\rm{Rec.}}}}}$ we combine the MAE loss and content loss

$${{{\mathcal{L}}}}_{{{\rm{Rec}}}}= {\lambda }_{{{\rm{mae}}}}\cdot ({{{\mathcal{L}}}}_{{{\rm{MAE}}},{{\rm{BAB}}}}+{{{\mathcal{L}}}}_{{{\rm{MAE}}},{{\rm{ABA}}}}) \\ +{\lambda }_{{{\rm{mae}}},{{\rm{id}}}}\cdot ({{{\mathcal{L}}}}_{{{\rm{MAE}}},{{\rm{BB}}}}+{{{\mathcal{L}}}}_{{{\rm{MAE}}},{{\rm{AA}}}}) \\ +{\lambda }_{{{\rm{content}}}}\cdot \frac{1}{4M}{\sum }_{j=1}^{M}\left({{{\mathcal{L}}}}_{{{\rm{Con}}},{{\rm{BAB}}},j}+{{{\mathcal{L}}}}_{{{\rm{Con}}},{{\rm{ABA}}},j}\right) \\ +{\lambda }_{{{\rm{content}}},{{\rm{id}}}}\cdot \frac{1}{4M}{\sum }_{j=1}^{M}\left({{{\mathcal{L}}}}_{{{\rm{Con}}},{{\rm{BB}}},j}+{{{\mathcal{L}}}}_{{{\rm{Con}}},{{\rm{AA}}},j}\right) \\ +{\lambda }_{{{\rm{noise}}}}\cdot ({{{\mathcal{L}}}}_{{{\rm{Noise}}},{{\rm{ABA}}}}+{{{\mathcal{L}}}}_{{{\rm{Noise}}},{{\rm{AA}}}}+{{{\mathcal{L}}}}_{{{\rm{Noise}}},{{\rm{BAB}}}}),$$

(7)

where M refers to the number of networks used for the discriminators and the λ parameters are used to scale the individual losses. The factor 4M is introduced to normalize the content loss for the number of discriminator layers per domain (cf. Eq. (3)).

As originally proposed by Goodfellow et al.⁶², GANs are composed of a generating network (generator) that produces a synthetic image from a random input vector (latent space), and a discriminating network (discriminator) that distinguishes between generated and real images. The training is performed in a competitive setup between the generator and discriminator. We optimize the discriminator D and generator G for the objective proposed by Mao et al.⁷⁶ (Least-Squares GAN). The discriminator A enforces the generation of low-quality images in the B-A-B cycle

$${{{{\mathcal{L}}}}}_{{D}_{{{{\rm{A}}}},j}}={\mathbb{E}}\left[{({D}_{{{{\rm{A}}}},j}({x}_{{{{\rm{A}}}}})-1)}^{2}\right]+{\mathbb{E}}\left[{D}_{{{{\rm{A}}}},j}{({G}_{{{{\rm{BA}}}}}({x}_{{{{\rm{B}}}}},z))}^{2}\right],$$

(8)

where D_{A, j} refers to the jth network of discriminator A. The generator is optimized to achieve a classification as real image by each discriminator network

$${{{\mathcal{L}}}}_{G,{{\rm{BA}}}}={\mathbb{E}}\left[\frac{1}{M}{\sum }_{j=1}^{M}{({D}_{{{\rm{A}}},j}({G}_{{{\rm{BA}}}}({x}_{{{\rm{B}}}},z))-1)}^{2}\right].$$

(9)

The loss for discriminator B and generator AB is computed in the same way, where the discriminator enforces the generation of high-quality images in the A-B-A cycle.

The variation of synthetic images is a frequently addressed topic for GANs^63,69,77. For conditional image translation a variation of generated images can be obtained by inducing additional noise terms^23,67 or by including style information^68,69. Instance normalization achieved significant improvements in neural style transfer⁷⁵. The normalization is applied after each convolution layer by normalizing each feature channel y for zero mean and unit variance

$$\,{{\mbox{IN}}}\,(y)=\gamma \frac{y-\mu (y)}{\sigma (y)}+\beta,$$

(10)

where μ(y) refers to the mean and σ(y) to the variance across the spatial dimensions and γ and β to learnable parameters.

The effect of instance normalization can be interpreted as a normalization of the feature space in terms of style, where the affine parameters (γ and β) act as a projection to a different style⁶⁸. In the present case, the translation between different instruments can be understood as style transfer, where we transfer the high-quality style to images of low-quality. The affine parameters allow the network to learn a single style. For the training with image patches, we use running estimates of the normalization variables (μ, σ) with a momentum of 0.01 (statistical parameters are computed according to μ_new = (1 − 0.01) × μ_old + 0.01 × μ_observed).

For the generator BA we enable the generation of various low-quality images from a single high-quality image by including a noise term, which we sample from a uniform distribution [0, 1]. A frequent problem of GANs is mode collapse, where the network generates the same image independent of the noise term^63,77. We introduce an additional loss term to prevent mode collapse and increase the diversity of generated low-quality images^69,78,79. We sample two independent noise terms (z and $\hat{z}$) for the same high-quality image x_B and compute the content loss between the resulting two images

$$\begin{array}{l}{{{\mathcal{L}}}}_{{{\rm{Con}}},{{\rm{Div}}},j}({x}_{{{\rm{B}}}},z,\hat{z})={\sum }_{i=1}^{4}\frac{1}{{N}_{i}}\left[\parallel {D}_{{{\rm{A}}},j}^{(i)}({G}_{{{\rm{BA}}}}({x}_{{{\rm{B}}}},z))-\right.\\ \left.{D}_{{{\rm{A}}},j}^{(i)}({G}_{{{\rm{BA}}}}({x}_{{{\rm{B}}}},\hat{z})){\parallel }_{1}\right].\end{array}$$

(11)

We scale the difference in content loss by the distance of the noise terms and apply the logarithm to the result, which leads to an increased loss for nearly identical images and reduces divergences for large differences

$${{{{\mathcal{L}}}}}_{{{{\rm{Div}}}}}={\mathbb{E}}\left[\log \frac{\frac{1}{4M}{\sum }_{j=1}^{M}{{{{\mathcal{L}}}}}_{{{{\rm{C}}}},{{{\rm{Div}}}},j}({x}_{{{{\rm{B}}}}},z,\hat{z})}{\parallel z-\hat{z}{\parallel }_{1}}\right].$$

(12)

The training cycles for the generators and discriminators are performed end-to-end, where we alternate between generator and discriminator updates. Our full generator objective is given by combining Eq. (7), (9) and (12)

$$\begin{array}{l}{\min }_{{G}_{{{\rm{AB}}}},{G}_{{{\rm{BA}}}},NE}\left({{{\mathcal{L}}}}_{{{\rm{Rec}}}}-{\lambda }_{{{\rm{divsersity}}}}\cdot {{{\mathcal{L}}}}_{{{\rm{Div}}}}+\right.\\ \left.{\lambda }_{{{\rm{adversarial}}}}\cdot \left({{{\mathcal{L}}}}_{G,{{\rm{BA}}}}+{{{\mathcal{L}}}}_{G,{{\rm{AB}}}}\right)\right),\end{array}$$

(13)

where the λ parameters are used to scale the individual losses. The discriminator objective is obtained from Eq. (8)

$${\min }_{{D}_{{{\rm{A}}}},{D}_{{{\rm{B}}}}}\left(\frac{1}{M}{\sum }_{j=1}^{M}({{{\mathcal{L}}}}_{{D}_{{{\rm{A}}},j}}+{{{\mathcal{L}}}}_{{D}_{{{\rm{B}}},j}})\right).$$

(14)

D_A and D_B refer to the combined discriminators for images of domain A and B, respectively (D_A = {D_A,1, . . . , D_A,M}).

Training parameters

For our model training we use the Adam optimizer with a learning rate of 0.0001 and set β₁ = 0.5 and β₂ = 0.9⁸⁰. As default value we set all λ parameters to 1. The content loss is scaled by 10 and the identity losses are scaled by 0.1 (s.t. λ_content = 10, λ_content,id = 1, λ_mae,id = 0.1). For all our models we track running statistics of the instance normalization layers (γ, β in Eq. (10)) for the first 100,000 iterations and fix the values for the remaining iterations. For all space-based observations we set λ_diversity = 0, such that the main focus is the adaption of the instrumental characteristics and the generation of spurious artifacts is neglected (e.g., pixel errors). The training is performed with batch size 1 and the size of image patches is chosen based on our computational capabilities.

In the Supplementary Table 2 we summarize the parameters of our model training. For each run we stop the training in case of convergence (no further improvement over 20,000 iterations). For the more complex low-quality features and limited number of samples, the generator BA can diverge to unrealistic low-quality images at later iterations. Here, we reduce the training iterations, where we note a stable training.

Evaluation

Pair-wise alignment

For comparing continuum observations from SDO/HMI and Hinode/BFI we perform a pixel-wise alignment of image patches. We only consider images that contain solar features (e.g., we avoid quiet-Sun and limb regions) and that have been recorded within a time frame of  <10 s. To this aim, we scale the Hinode/BFI observations to 0.15 arcsecs and the SDO/HMI observations to 0.6 arcsecs per pixel. We apply our ITI method to upsample the HMI images by a factor of 4 in order to match the Hinode/BFI observations. From the header information of the Hinode/BFI files we select the corresponding HMI region. We co-register the translated HMI patches with the Hinode/BFI patches using the method by⁵¹, that determines translational shifts and rotational differences, based on phase-correlation of the Fourier-transformed images. After the co-alignment we truncate the edges (80 pixels per side) to avoid boundary artifacts. As baseline image enhancement method we apply a Richardson-Lucy deconvolution⁴⁷, using the estimated point-spread-function of HMI by Yeo et al.⁴⁸ (K₁), and applying 30 iterations. The deconvolved images are upsampled by a factor 4 using bilinear interpolation and co-aligned analogously to the ITI image patches.

To compare the pixel counts I between Hinode/BFI and deconvolved HMI images we analogously extract patches from the training set and compute the mean ($\overline{I}$) and standard deviation (σ) over the full data set, which is used to adjust the HMI intensities as

$${I}_{{{{\rm{calibrated}}}}}=({I}_{{{{\rm{A}}}}}-\overline{{I}_{{{{\rm{A}}}}}})\cdot \frac{{\sigma }_{{{{\rm{B}}}}}}{{\sigma }_{{{{\rm{A}}}}}}+\overline{{I}_{{{{\rm{B}}}}}},$$

(15)

where A refers to the source calibration (HMI) and B to the target calibration (Hinode).

For the comparison between SOHO/EIT + MDI and SDO/AIA + HMI we apply a similar alignment procedure, where we apply the spatial image registration to the full-disk observations. Transformations of the full-disk observations are mostly rotations, while the disk is typically sufficiently well centered. For the paired samples of KSO observations of low- and high-quality, we perform a more extensive co-alignment to address changes due to solar rotation and atmospheric effects (e.g., image blurring). We first perform a differential de-rotation to the reference observation. From the resulting image we extract the central 512 × 512 pixels to truncate the off-limb region, which we do not consider for our evaluation. Finally, we co-register the image patches using the method from ref. ⁵¹.

Calibration

As calibration baseline we assume that the quiet-Sun is invariant to solar-cycle variations. We extract quiet-Sun regions by selecting all on-disk pixels that are below the estimated threshold. To this aim, we compute the mean and standard deviation per channel from observations sampled across the timeline and set the threshold to the mean + 1σ. For each data set, we compute the mean and standard deviation of the quiet-sun pixels. For our baseline calibration we adjust the mean and standard deviation between the channels of the individual instruments using Eq. (15).

Metrics

For the pixel-wise comparison between paired samples we use three primary metrics. (1) The Structural Similarity Index (SSIM;⁸¹). (2) The Peak signal-to-noise ratio

$${PSNR}=10\, {\log }_{10}\left(\frac{{I}_{\,{\mbox{max}}\,}^{2}}{{MSE}}\right),$$

(16)

where I_max refers to the maximum intensity or value range of the data set, and MSE to the mean-squared-error. (3) The image cross-correlation (CC)

$$C{C}_{xy}=\frac{\mathop{\sum }_{n=0}^{N-1}({x}_{n}-\bar{x})({y}_{n}-\bar{y})}{{\sigma }_{x}{\sigma }_{y}},$$

(17)

where x and y refer to the paired images, N to the total number of pixels, n to the pixel index in the image, and σ to the respective standard deviation.

For the computation of the FID, we convert the test sets of each dataset to gray-scale images. Differences in resolution between high- and low-quality data sets, are adjusted by bilinear upsampling. The reconstructed STEREO magnetograms are not included in this evaluation, since there exists no low-quality data set for comparison (see, “Methods” subsection “Datasets”). We use an InceptionV3 model with the weights of the tensorflow implementation for the FID. The FID is evaluated for each application and channel separately, where we use the full resolution images (i.e., no resizing) and do not normalize the inputs.