High-resolution Solar Image Reconstruction Based on Non-rigid Alignment

Hui Liu , Zhenyu Jin, Yongyuan Xiang, and Kaifan Ji

Yunnan Observatories, Chinese Academy of Sciences, Kunming 650216, China; liuhui@ynao.ac.cn

Abstract Suppressing the interference of atmospheric turbulence and obtaining observation data with a high spatial resolution are an issue to be solved urgently for ground observations.One way to solve this problem is to perform a statistical reconstruction of short-exposure speckle images. Combining the rapidity of Shift-Add and the accuracy of speckle masking, this paper proposes a novel reconstruction algorithm-NASIR (Non-rigid Alignment based Solar Image Reconstruction). NASIR reconstructs the phase of the object image at each frequency by building a computational model between geometric distortion and intensity distribution and reconstructs the modulus of the object image on the aligned speckle images by speckle interferometry.We analyzed the performance of NASIR by using the correlation coefficient,power spectrum,and coefficient of variation of intensity profile in processing data obtained by the NVST (1 m New Vacuum Solar Telescope). The reconstruction experiments and analysis results show that the quality of images reconstructed by NASIR is close to speckle masking when the seeing is good,while NASIR has excellent robustness when the seeing condition becomes worse.Furthermore,NASIR reconstructs the entire field of view in parallel in one go, without phase recursion and block-by-block reconstruction, so its computation time is less than half that of speckle masking. Therefore, we consider NASIR is a robust and highquality fast reconstruction method that can serve as an effective tool for data filtering and quick look.

Key words: methods: data analysis – techniques: image processing – Sun: chromosphere – Sun: photosphere –instrumentation: high angular resolution

1. Introduction

At present, solar physics research has entered a period of small scale and fine structure. High-resolution observational images are urgently needed. However, diffraction-limited imaging cannot be achieved with ground-based optical telescopes due to the interference of the turbulent atmosphere.

To mitigate the adverse effects of the atmosphere and achieve near-diffraction-limited imaging, the researchers propose the speckle imaging method for high-resolution reconstruction by statistical analysis. The speckle imaging method started from the pioneering work of Labeyrie (1970), and its processing object is the speckle images. Speckle images are short-exposure images taken with a telescope with an aperture much larger than the atmospheric coherence length. The speckle imaging method can obtain high-resolution reconstructed images close to the diffraction limit of the telescope and has a relatively complete theoretical support.

Speckle imaging methods can roughly be classified into spatial domain and frequency domain reconstruction methods.Spatial reconstruction methods directly reconstruct the spatial structure and intensity of the object. The characteristics of the spatial domain method are simple algorithms and clear spatial descriptions. The typical representative of the spatial domain reconstruction method is Shift-Add (Bates & Cady 1980).Frequency domain reconstruction methods usually reconstruct the phase and modulus separately. Phase is reconstructed by speckle masking (Lohmann et al. 1983), while modulus is by speckle interferometry (Labeyrie 1970). Most of the abovementioned classic algorithms originated in the 1970s and 1980s, and subsequent researchers also have substantial and valuable research works to improve these methods (Liu et al.1999; Jin et al. 2008; Huo & Zhou 2010; Xiang et al. 2016).However, there are still issues that need to be further studied and resolved in practical applications.

The main issue of spatial domain methods is to align the regions containing object diffraction imaging information accurately. Taking Shift-Add as an example, the algorithm assumes that the maximum intensity points of the speckle images coincide with that of the object image and performs rigid translation(uniform displacement of the entire image)and superimposition of each speckle image to obtain a reconstructed image.However,for an extended source like Sun,due to the influence of the surrounding structure,the distribution of maximum intensity points will change significantly under the influence of the atmosphere. Not only exist multiple points of maximum intensity, but their positions will also deviate. The assumption that points of maximum intensity coincide each other no longer holds. In addition, the overall displacement of the image ignores the distortion differences caused by different fields of view and frequency components, which makes it difficult to obtain satisfactory reconstruction results with the traditional ®rigid-Shift-Add.

Speckle imaging frequency-domain reconstruction algorithm is currently the most widely used high-resolution reconstruction method (Liu et al. 1999). Well-known ground-based solar telescopes such as NVST (1m New Vacuum Solar Telescope)(Liu et al.2014)and GST(Goode Solar Telescope)(Cao et al.2010) use this method to reconstruct observational data. A typical representative of the frequency domain reconstruction method is speckle masking.

Because speckle masking estimates the phase of the object based on the triple correlation of the speckle images, that is,recursively from the low-frequency phase to the highfrequency, it is inevitably affected by noise such as photon noise during the recursion process. Due to the recursion process, the noise influence will accumulate in the highfrequency part, and in severe cases, it will distort the fine structure of the object. Therefore, the performance of speckle masking will significantly reduce with the decrease in seeing(Liu et al. 1999).

Therefore, we propose a new high-resolution solar image reconstruction algorithm named Non-rigid Alignment based Solar Image Reconstruction (NASIR) in this paper. NASIR draws on the basic theoretical models of speckle masking and Shift-Add and combines the ideas of the optical flow method(Horn & Schunck 1981) in computer vision. The algorithm obtains the distortion displacement field of each frame of the speckle image relative to the reference image (RI) by establishing a computational model between the intensity change and the displacement field. The distortion is corrected by pixel-by-pixel non-rigid alignment to obtain the phase estimation of the object image. The modulus of the object image is by speckle interferometry on the aligned speckle images.

Section 2 of the paper introduces the principle and realization of NASIR. Section 3 applies the algorithm to reconstruct the observation data of NVST and compares it with the classical methods. Section 4 discusses the features of NASIR and issues that need to be further studied. Our conclusions are provided in Section 5.

The above process involves displacement field estimation,bandpass filtering, iterative alignment, and modulus reconstruction of the object image.

2.1. Displacement Field Estimation Based on Intensity Changes

Based on the idea of the optical flow, we established the pixel displacement and intensity variation model of the speckle image. With this model, the phase distortion caused by atmospheric turbulence is corrected by reverse displacement correction.For the convenience of analysis and calculation,we first use a polynomial to approximate the relationship between pixel intensity and pixel coordinates in a local region of the image. Taking the quadratic polynomial as an example, the intensity distribution of the local area in a speckle image can express approximately as:

Here x is the coordinates of a local region in a speckle image,f(x) is the pixel intensity at x, A is a symmetric matrix, b is a displacement vector,and c is a scalar.These parameters can be estimated by the weighted least square method.

Assume that the approximate expression formula of the corresponding region of the reference image RI (This article uses the ensemble average of the speckle images as the initial reference image) is:

If the speckle image has an ideal translation d with respect to RI in this region,the relationship between the intensity of this area and the spatial coordinate of the speckle image can be expressed as:

2. The Principle and Implementation of NASIR

Similar to speckle masking, NASIR also reconstructs phase and modulus separately. For phase reconstruction, NASIR obtains the distorted displacement field of each frame of the speckle image relative to the reference image (RI) by establishing a computational model between the intensity variation and the displacement field. The phase estimation of the object image is by correcting the distortion by pixel-bypixel non-rigid alignment. The modulus is by speckle interferometry on the aligned speckle images.

In the non-singular case of A1,we can obtain the translation d through Equation (5):

So we established a computational model between pixel intensity and displacement under ideal translation. To more accurately describe the projection of the distortion on the image, a reasonable model of the displacement field can be obtained through optimization strategies such as the affine transformation of the displacement field and the weighting of neighborhood pixels Farnebäck(2003).The specific method is as follows: for the projection of the general three-dimensional motion on the image plane, we use the affine transformation model to express the displacement field as:

Here i is the pixels in the neighborhood,wiis the weights of the pixel i.Neighborhood size needs to determine according to seeing (r0) (which will discuss in Section 4.2). Solving P in Equation (13) and substituting it into Equation (10), we can obtain the displacement vector d of each pixel relative to RI.Each speckle image can align to RI by inversely shifting and interpolating each pixel according to the d calculated at each time.

2.2. Band-pass Filtering

Due to the properties of atmospheric turbulence,the original speckle image is dominated by low-frequency energy. The displacement field calculated directly from the original speckle image mainly reflects the distortion of the low-frequency structure,which is not conducive to the recovery of the mid and high-frequency phase. Therefore, based on the theoretical speckle interference transfer function (sitf ) Korff (1973) and the optical transfer function(Otf)of NVST Xiang et al.(2016),we constructed a bandpass filter Hbp(u) as shown in Equation (14) to filter the original speckle image. This filter enhances mid and high-frequency features and suppresses noise outside the diffraction limit, thereby improving the quality of image alignment.

Here Snis the noise power spectrum estimated from the speckle image sequence, and u is the spatial frequency.

2.3. Reference Image Update and Iterative Alignment

Estimating the distortion displacement requires an image with a higher signal-to-noise ratio as the reference image RI(described in Section 2.1). The initial RI is the ensembleaverage image of the speckle images filtered by Equation(14).After the speckle images align to the current RI, the signal-tonoise ratio of their ensemble-averaged image is improved, at this time, NASIR updates the RI with the newer ensembleaveraged image and performs displacement field estimation and alignment again.

After multiple iterations of alignment and updating, the quality of RI and the alignment accuracy of speckle images continuously improved. Then the phase of the ensembleaveraged image of the speckle images is used as the phase estimation of the object image.

2.4. Modulus Reconstruction Based on Speckle Interferometry

The basic idea of speckle interferometry is that when the exposure time is less than a certain threshold, the speckle images contain more information than the long-exposure image. The power spectrum of multiple consecutive short exposure speckle images can be time-averaged to restore the power spectrum of the object image Labeyrie (1970).

In practical observation, speckle images often contain various random noises. Therefore, NASIR uses the Wiener filter method of Equation(15)to reconstruct the modulus of the object image:

where |O(u)| is the modulus of the object image, 〈|Ii(u)|2〉is the ensemble-average of the power spectrum of the speckle images, Snis the noise power spectrum estimated from the speckle image sequence, sift(u) is calculated based on the seeing Korff (1973).

Since the distortions in speckle images are corrected, their ensemble-average images contain more mid-and high-frequency energy, which can further improve the contrast of the reconstructed image. So, unlike speckle masking, NASIR uses the aligned speckle image instead of the original speckle image to calculate〈|Ii(u)|2〉and then obtains |O(u)| from Equation (15).

3. Reconstruction Results and Analysis

We compare the reconstruction performance of NASIR with that of correlation Shift-Add and speckle masking by reconstruction experiments on NVST observations.

NVST is the largest ground-based vacuum solar telescope in China, and it was put into use in 2010. The NVST has a clear aperture of 985 mm and is mainly used for high-resolutionimaging observations of the solar photosphere, chromosphere,and spectrum(Liu et al.2014).The main instruments of NVST include high-resolution multi-channel imaging systems (Xu et al. 2013) and high-dispersion multi-wavelength spectrometers (Wang et al. 2013).

Table 1 Speckle Images High-resolution Reconstruction Test Data set

NVST data products are divided into three levels:Level 0 is the original observation data; Level 1 is based on lucky imaging, corrected by dark current and flat field, and reconstructed by the Shift-Add; Level 1+ is reconstructed by speckle masking, which can produce reconstruction results close to the diffraction limit (Yan et al. 2020).

The experimental data set in this paper is derived from the Level 0 data of the NVST photosphere(TiO,center wavelength 705.8 nm, bandwidth 1 nm) and chromosphere (Hα, center wavelength 656.28±0.4 nm, bandwidth 0.025 nm) under different seeing degrees from 2013 to 2019. Each set of experimental data contains 100 short-exposure speckle images(Table 1). The seeing parameter r0is calculated from the spectral ratio method (von der Lühe 1984).

3.1. Analysis of Reconstruction Quality When Seeing is Good

Figure 1 shows the reconstruction results of NVST TiO observation data (data set No.1 in Table 1) using correlation Shift-Add, speckle masking, and NASIR, respectively.

It can be seen from Figure 1 (granule outlines, bright point and penumbral filaments of sunspot,etc.)that the reconstruction quality of NASIR is significantly better than that of Shift-Add,and it is closer to the reconstruction quality of speckle masking.

To intuitively compare the reconstruction quality in the frequency domain,we convert the 2D power spectrum image to polar coordinates. Then, the power spectrum image in polar coordinates is averaged along the polar angular to obtain a 1D power spectrum curve with the length of the polar diameter as the frequency and the average intensity as the energy.

The power spectrum curve in Figure 2 (left) shows that the resolution of the NASIR reconstructed image is close to that of speckle masking reconstructed image, about 0.15 arc-second,which is significantly higher than that of Shift-Add reconstructed image of about 0.25 arc-second. The intensity profiles in Figure 2 (right) further show the similarity in intensity distribution between NASIR and speckle masking reconstruction.

To further reveal the performance of NASIR, we use the following quantitative metrics to compare and analyze the reconstruction results of speckle masking and NASIR:

(1) Image correlation coefficient (r): The correlation coefficient is used to compare the degree of linear correlation between reconstructed images, which is defined as:

Here, Cov(X, Y) is the covariance of the pixel intensity of the two images X and Y,Var[X]and Var[Y]are the variances of X and Y respectively.

(2) Structural SIMilarity (SSIM) (Wang et al. 2004): SSIM is an index that measures the similarity of two images. Given two images X and Y,their structural similarity can be defined in the following way:

where μxis the mean of X,μyis the mean of Y,σxis the variance of X,σyis the variance of Y,and σxyis the covariance between X and Y. c1=(k1L), c2=(k2L) are constants used to maintain stability. L is the range level of pixel values. k1=0.03,k2=0.03. The range of structural similarity is -1 to 1. When the two images are the same, the value of SSIM is equal to 1.

The structural similarity index defines structural information from the perspective of image composition as being independent of brightness and contrast, reflecting the properties of the object structure in the scene, and modeling distortion as a combination of three different factors,brightness,contrast,and structure. The mean is used as an estimate of brightness, the standard deviation is used as an estimate of contrast, and covariance is used as a measure of structural similarity.

(3)Coefficient of Variation of Intensity Profile(CVoIP):We use the coefficient of variation of the intensity profile of the reconstructed image, which is the ratio of the standard deviation of the intensity profile to its mean, to measure the difference in image contrast.

A quantitative comparison of speckle masking with NASIR is shown in Table 2,indicating that the reconstructed images of the two methods have high correlation and structural similarity.Therefore, the visual quality (Figure 1), the consistency of the power spectrum curve (Figure 2), and the quantitative comparison index of Table 2(first row)show that when seeing is good,NASIR can achieve reconstruction quality close to the speckle masking method.

The reconstruction results of the Hα-band data further verify the effectiveness of the NASIR for high-resolution reconstruction of the speckle images when the seeing is better. Figure 3 shows the reconstructed images of the three algorithms for data set 2 in Table 1.The quantitative comparison result is in the second row in Table 2. The power spectrum curves and the intensity profile curves are shown in Figure 4.

Figure 1.Reconstruction results of the NVST TiO observations with the active area 11 801,2013-08-01T:03:44:53.(a)A frame in a sequence of speckle images.(b)Reconstruction by correlation Shift-Add. (c) Reconstruction by speckle masking. (d) NASIR reconstruction.

3.2. Comparison of Reconstruction Performance with Slightly Poor Seeing

Figure 5 shows the reconstruction results of data set 3(r0=5.88 cm).It can be seen that as the seeing decreases,the recursive cumulative error of the phase of speckle masking has an aggravated influence on the reconstruction, resulting in stray structures in the reconstructed image, while NASIR can better maintain the characteristics of granules. The reconstruction results of the two algorithms still have high correlation and structural similarity, but the CVoIP values reflect the higher contrast of the reconstructed images from NASIR. The third row of Table 2 gives the comparative quantitative indicators.

Figure 2. Power spectrum curves (left) and intensity profiles (right, the red line region in Figure 1(a)) of images in Figure 1.

Table 2 Quantitative Comparison of the Reconstruction Results of the Four Groups of Test Data (Table 1)

Figure 6 shows the reconstruction results of data set 4(r0=4.09 cm) in Table 1. As we can see, with the further decrease in seeing,the correlation coefficient r and SSIM of the images reconstructed by the two algorithms start to decrease significantly (the fourth row of Table 2), indicating a clear difference in the reconstructed images. The phase recurrence error of speckle masking seriously affects the reconstruction quality, the reconstructed image has been destroyed by the pseudo-structure, while NASIR can still obtain low-frequency structures to a certain degree.

3.3. Computational Time-consuming

NASIR is a spatially parallel algorithm that avoids the multipath phase recursion,block reconstruction,and stitching,so its average computation time is about 48.7% of speckle masking.

4. Discussion

The core idea of NASIR is to recover the phase distortion of different frequency components through non-rigid (nonlinear)geometric correction. It is a fast and robust method for highresolution statistical reconstruction.

4.1. Features of NASIR

ASIR performs phase correction through pixel-by-pixel nonrigid alignment, which is the affine correspondence between pixel feature descriptors. Feature descriptors are polynomial fit coefficients of the intensities of pixels and their neighborhoods.So non-rigid alignments are not limited by spatial invariance(isoplanatic angle). In addition, the continuous update of the reference image RI by NASIR can make the low-frequency structure tend to be consistent during the iterative alignment process, so mitigating the geometric distortions between isoplanatic angles. Therefore, NASIR can directly reconstruct the field of view at a time without block-wise reconstruction.

NASIR avoids the computation of high-order statistics and complex multi-path phase recurrence,and the computation time is only about half that of speckle masking.

NASIR estimates the displacement field for each pixel of the field of view in parallel, which can be immune to recursive error accumulation.

Band-pass filtering of the raw speckle images suppresses noise interference beyond the diffraction limit, allowing NASIR to reconstruct the modulus and phase of the midfrequency structure more efficiently.

4.2. Further Improvements to NASIR

While achieving successful experiments, NASIR deserves further optimization and improvement in the following areas:

(1) Optimization of displacement field-intensity distribution model. Considering the convenience of analysis and calculation,NASIR adopts polynomial expansion to approximate the intensity distribution of the image area. It may differ from physical models of actual intensity changes caused by atmospheric turbulence. Therefore, establishing a speckle image intensity distribution-displacement representation model more in line with physics is one of the issues worthy of further study.

Figure 3. Reconstruction results and power spectrum of NVST Hα 2019-10-04T02:59:05 (active area 12 749).

(2)Adaptive selection of neighborhood size.NASIR uses the intensity distribution model of the neighborhood as a feature descriptor and estimates the displacement through the matching between feature descriptors. A larger neighborhood size can improve the matching stability, but it will produce more smoothing effects and reduce the reconstruction quality of high-frequency structures. Conversely, a smaller neighborhood size can preserve high-frequency details.But in the case of poor vision and high noise, it is easy to cause mismatch and reduce the accuracy of displacement estimation. Therefore, a more appropriate size should be chosen according to the atmospheric coherence length for better alignment performance.

(3) Design and implementation of seeing-based bandpass filters. NASIR uses bandpass filtered speckle images for alignment, essentially phase correction based on the alignment of object structures within the diffraction limit. Therefore,seeing-based adaptive bandpass filter design is also one of the fundamental issues to consider.

Figure 4. Power spectrum curves (left) and intensity profiles (right) of images in Figure 3.

Figure 5. Comparison of reconstruction results when r0=5.88 cm, (a) speckle masking, (b) NASIR.

(4) Improve alignment and update iteration strategies. Nonrigid alignment based on speckle imaging requires a reference image. Iterative improvement of reference image quality is crucial for phase reconstruction. Empirically, the number of iterations for NASIR to update the reference image is between 3 ～5 times,therefore,it is necessary to make more reasonable choices and judgments on iteration and convergence based on seeing r0.

5. Conclusion

NASIR is a novel high-resolution reconstruction method for speckle imaging that combines classical statistical reconstruction methods and computer vision techniques. Reconstruction results from ground-based telescope observations demonstrate the potential of NASIR in several ways:

Flexibility. NASIR can directly reconstruct the full field of view without sub-block reconstruction and stitching, avoiding stitching errors and discontinuities between stitched sub-blocks.

Figure 6. Comparison of reconstruction results when r0=4.09 cm, (a) speckle masking, (b) NASIR.

Robustness.When seeing is good,the reconstruction quality of NASIR is close to speckle masking.When seeing decreases,NASIR is more stable.

Fastness. NASIR does not perform phase recursion and avoids block reconstruction/stitching operations. Computation time is reduced by more than 50% compared to speckle masking.

In conclusion, although NASIR does not have theoretical accuracy guarantees like speckle masking. However, experimental results show that NASIR is still a convenient, robust,and fast method for high-resolution solar image reconstruction.We believe that it will play an important role in many applications such as quick look and data filtering.

Acknowledgments

We would like to thank the NVST team for high cadence data support. This work is sponsored by the National Natural Science Foundation of China (NSFC) under Grant Nos.11873027, U2031140, 12073077, 11833010 and 11973088,and West Light Foundation of the Chinese Academy of Sciences (Y9XB01A and Y9XB019).

ORCID iDs

Hui Liu https://orcid.org/0000-0003-2714-6811

Kaifan Ji https://orcid.org/0000-0001-8950-3875