SUPER RESOLUTION FOR EGYPTSAT-1 IMAGES WITH ERRATIC SHIFT

The key point of the Super-Resolution (SR) process is the accurate registration of the low resolution images, i.e., accurate measuring of the fixed shift between them, to obtain high resolution image. Due to cert ain malfunction, some Egyptsat-1 images have inconsiste nt ub-pixel shift. Therefore, in this study we pro pose a methodology to use this kind of shift for reconstru c ing a SR image of Egyptsat-1 from its low resolut i n bands. It is a trade-off between the capability of catching spatial details and the sensitivity to the erratic shift existed along the image. Firstly, this inconsistent shift between the bands is transformed into reliab le shift. Then a SR method based on image fusion scheme with multi-resolution decomposition is performed. The fusion process is conducted in steerable wavelet do main using normalized convolution technique. It all ows the recognition of objects with size approaching its li miting spatial resolution. Results show that the pr oposed methods make significant spatial resolution improve ments from 7.8 to 4 m. Different quantitative measu re of the proposed methodology were assessed and tested w ith some implemented commonly used SR methods. These methods are; nonparametric bayesian, POCS, it erative-interpolation, robust and iterated back pro jection. The visual and quantitative evaluations verify the us fulness and effectiveness of the proposed method ology.


INTRODUCTION
Accurate registration of images is the most fundamental component to high performance image processing techniques such as multi-frame image fusion, change detection and super-resolution. The everincreasing demand for more pixels, or higher resolution, in combination with the availability of more and more computational power, has generated a large interest in super-resolution imaging (Dewalle, 2006). Fusion algorithms combine one or several Low Resolution (LR) images with one or more High Resolution (HR) images in order to obtain a useful final image with better spatial resolution than LR, but maintaining as much spectral or temporal resolution as possible. In contrast, super-resolution algorithms do not use any HR image. They only combine independent LR images from the same scene and the only HR images shown are results of the methods (Merino and Nuñez, 2007).
The task of image SR requires an availability of low resolution observations of a scene. These observations can be captured simultaneously or at different times by a single or multiple imaging devices. If they have sub-pixel shifts then they contain additional information that can be used for high resolution reconstruction (Chaudhuri and Manjunath, 2005). The general strategy that characterizes SR comprises three major processing steps: LR image acquisition, image registration or motion estimation and HR image reconstruction (Stathaki, 2008). Image registration is a crucial intermediate step in

JCS
all image analysis tasks in which the final information is gained from the combination of various data sources. A comprehensive survey of image registration methods are given by (Zitova and Flusser, 2003;Brown, 1992). Applications of super-resolution both in civilian and military domain are; remote sensing, medical and Radar imaging, surveillance systems, target detection and recognition, to name a few (Borman, 2004). They have been applied to remote sensing images such as landsat, système pour l'Observation de la terre, QuickBird, hyperspectral and compact high resolution imaging spectrometer images (Chan et al., 2010). A broad review and explanation of development of the superresolution methods are given by (Borman and Stevenson, 1998;Park et al., 2003;Farsiu et al., 2004). In this study, we propose a methodology to reconstruct a SR image from the inconsistent sub-pixel shift of Egyptsat-1 bands and tested it with some implemented commonly used SR methods.
The remainder of the study is organized as follows: Section-1 describes the idea of the image registration, the steerable wavelet transform and the normalized convolution technique. Where section 2 focuses on the data acquisition and the study area. The proposed methodology and the results are discussed in sections 3 finally, the concluding remarks are given in section 4.

Image Registration
To formally define image registration, it is the process of overlaying two or more images of the same scene taken at different times, from different viewpoints and/or by different sensors (Zitova and Flusser, 2003). The accuracy of SR algorithms is quite often limited by the ability to register a set of lowresolution images (Robinson et al., 2009). In image registration, all the input images with irregularly spaced pixels need to be correctly aligned with each other on a common grid. Therefore, accurate knowledge of registration parameters is required for each of the input LR frames. Very high accuracy, i.e., precise alignment, is essentially required in the registration (up to sub-pixel level) to be able to reconstruct the HR image correctly. Miss-alignment of the LR frames will result in the reconstruction of an erroneous HR image. In general, we estimate the relative displacements of pixels between the different images with fractional pixel accuracy and then we combine together sampling points according to the estimated displacements, to produce a single image plane.

Steerable Wavelet Transform
The wavelet transform has become a popular image representation in recent years. It is being used increasingly for the processing of images (Starck and Pantin, 1996). It is based on the decomposition of the image into multiple channels, on the basis of their local frequency content, each with a different degree of resolution (i.e., multi-scale and multi-orientation components). The wavelet representation is an intermediate representation between the Fourier and the spatial representation. It is commonly implemented as a cascading series of high-pass and low-pass filters, based on the mother wavelet, applied sequentially to the lowpass image of the previous recursion. In practice, three or four recursions are sufficient. The conventional wavelet transform is neither shift-invariant nor rotation-invariant which is considered as a source of mismatches. Therefore, in this research we have used the Steerable wavelet transform introduced by Freeman and Adelson (1991) which provides a useful front-end for imageprocessing and computer vision applications (Portilla and Simoncelli, 2000). The steerable wavelet decomposition is a wavelet-like transform that is invariant to shift and rotation (Hel-Or and Keren, 2002). It is an over-complete representation having the following properties.
Its basis functions are shifted, scaled and rotated versions of a "mother wavelet"; If M(x, y) is a mother wavelet, then each basis kernel W s,d can be constructed by: are known interpolation functions. This is the "steerability" property of these filters. More details can be found in the reference (Freeman and Adelson, 1991).

Normalized Convolution
Fusion process and interpolation are usually done by convolution. The convolution can be made more effective by a normalization operation that takes into account the possibility of missing or unreliable samples (e.g., due to bad sensors or erroneous registration). Knutsson and Westin (1993) considered the problem of the image analysis when performed on irregularly sampled image data under the theory of signal and its certainty. This is to consider the separation of both data and operator applied to the data in a signal part and a certainty part. Missing data in irregularly sampled series is handled by setting the certainty of the data equal to zero. In the case of uncertain data, an estimate of certainty accompanies the data and this can be used in a probabilistic framework. The theory is called Normalized Convolution (NC). It is a technique for local signal modeling from projections onto a set of basis functions. Although any basis can be used, the most common one is a polynomial basis: {1, x, y, x 2 , y 2 , xy . .
· · · x N 2 ] T and so on are constructed from local coordinates of N input samples. The use of polynomial basis functions makes the traditional NC equivalent to a local Taylor series expansion. Within a local neighborhood centered at s 0 = {x 0 , y 0 }, the intensity value at position s = {x + x 0 , y + y 0 } is approximated by a polynomial expansion:

Data Acquisition and the Study Area
Egyptsat-1 is the first Egyptian earth observation satellite which was successfully put in orbit on 17/04/2007. It is a micro-satellite technology built in cooperation with the Ukraine. It plays a vital role as a source of information for the remote sensing and GIS communities (Nasr and Helmy, 2009). It captures both multi-spectral and panchromatic images. The characteristics of its data are as follows: • The spatial resolutions of the Multi-spectral and the panchromatic bands are 7.8 meter • The spatial resolution of the Mid-Infrared band is 39.5 meter • The spectral resolutions are as shown in Table 1 Due to malfunctioning of Egyptsat-1sensor, there is an irregular shift between band 1 and band 2 in some parts of its images (Nasr and Helmy, 2011). We are going to use this shift for the production of the super resolution image.
The study area is delimited geographically by latitude 30° 06' N and 30° 04' N and longitudes 31° 18' E and 31° 20' E approximately. This area is located in the western part of Cairo. Subsets of bands 1 and 2 of the Egyptsat-1 image covering this area and acquired on 28/7/2009 has been used in conducting our research, as shown in Fig. 1.

MATERIALS AND METHODS
proposed methodology to transform the inconsistent sub-pixel shift into a reliable one is described for efficiently registering a set of LR images (Egyptsat-1 bands). It has been reported that band 1 and band 2 are misaligned, producing unregistered separate bands. The shift between these two bands is not constant along the whole image. Therefore, the basic idea behind the following procedures is to change the inconsistent shift between the two bands into a constant shift and then use this shift to produce super resolution image with merged spectral information from both bands. In order to overcome the inconsistency of the sub-pixel shift through the LR bands, we propose the following method Nasr and Helmy, (2009) in the block diagram shown in Fig. 2. The main steps of the method can be summarized as follows: • Divide the input LR bands into small patches, N block of size M×M • Construct an error surface between these two bands by computing the shift in x, y directions for each block • Use joint frequency distribution analysis to determine an optimal x and y shift in order to maximize reconstruction accuracy while minimizing the overall shift error • Project one of the input LR bands into an empty grid shifted by delta x, y derived from the previous step Applying these steps yield derived LR band 2 with constant shift with respect to band 1. Our hypothesis about this proposed method is based on the following assumption; the shift between any two LR images, that occupies the largest area, will produce better super resolution image if it is widespread all over the image (i.e., making this shift constant between the two images). To prove the concept, we begin our analysis by using two synthetic images with 0.5 pixel artificial shift, as shown in Fig. 3. In measuring the shift in the x and y directions between the two images, we divided them into (50×50 pixel) blocks. The number of repetition (histogram) of the shift is calculated and plotted separately as shown in Fig.  4a. They are also represented in 3D as shown in Fig. 4b. Then, a crude statistics of the "shift" was derived, ranging from 0 (no shift) to 9 (one pixel shift), in the x and y direction as summarized in Table 2 and 3 respectively.
It is very clear from the above two tables that the most frequently occurred shift in the x and y direction is the 0.5 pixel shift. When it is used to reconstruct the SR image, as shown in Fig. 5, it proves the concept and confirms our proposed method. Applying this procedure on the Egyptsat-1 bands, we divided them into (40×40 pixel) blocks. The results of the frequency Science Publications JCS distribution of the shift in the x and y directions are calculated (Vandewalle et al., 2004) and plotted separately as shown in Fig. 6a and the 3D representation is shown in Fig. 6b.   Fig. 3. Two synthetic images with 0.5 pixel artificial shift    To reconstruct a SR image (i.e., generate a single 4 m pixel image from the previous two subsets 7.8 m pixel bands), Matlab implementations have been used. We applied the steerable wavelet decomposition algorithm to overcome the limitation of the orthogonal separable wavelet transform and used the normalized convolution technique for the SR image reconstruction, as depicted in the following steps of the block diagram shown in Fig. 7.
The main steps of the algorithm are: • High Pass Filtering (HPF) the second (shifted) Egyptsat-1 image band 2 • Decomposing of the LR input band 1 and the HPF of band 2 into their wavelet planes in the steerable wavelet domain • Convolving their outputs using the normalized convolution technique • Reconstructing the SR output image using the inverse decomposition process Since the spectral characteristics of bands 1, 2 are different and to overcome the modification of radiometry (i.e., limit the spectral distortion) of the output SR image after fusion, we used the High-Frequency Component (HPF) of band 2. The most detailed scale information of the image is located in the first wavelet planes, therefore those are the fused planes from the LR and the HPF included in the resulting SR image. We produced a single 4 m pixel panchromatic image of 2660×2940 pixels from the 7.8 m pixel images of 1330×1470 pixels. Figure 8 shows the SR reconstructed image of the Egyptsat-1 data. In Fig. 9 parts of band 1, the shifted band 2 and the SR image are shown zoomed enough to depict the improvement in spatial resolution. . 9. The images are zoomed to depict the improvement in spatial resolution, (a) Band 1

RESULTS AND DISCUSSION
By visual comparison of the original LR images and the reconstructed SR output, it is easy to see that the spatial resolution of the SR image is clearly better than the original ones (i.e., increased). Thus, the proposed methodology and the used algorithms produce a significant improvement. Visually the SR image contains more information than the original bands. Objects as different as streets, green areas and buildings are clearly better shaped and/or recovered.
Besides the evident improvements shown in Fig. 9, we were also interested in an objective evaluation of the results. We have used five quantitative measures: The mutual information I AF (Wei and Jing, 2007;Wang and Bovik, 2002), Q AB/F (Xydeas and Petrovic, 2000), Q 0 (Zheng et al., 2007), Q W (Zheng et al., 2007;Qu et al., 2002) and Q E (Piella and Heijmans, 2003) metrics which have been proved to be effective to a great extent.
The mutual information I AF reflects the statistical dependence of two random variables as it measures the similarity of image intensity distribution between the source image A and the SR image F. It is defined as follows: where, P AF is the jointly normalized histogram of A and F. P A and P F are the normalized histogram of A

JCS
and F and (a) and (f) represent the pixel value of the image A and F, respectively. The mutual information I BF between the source image B and the super resolution image F is similar to I AF . The mutual information between the source images A, B and the SR image F is the sum of I BF and I AF , i.e.: The metric Q AB/F evaluates the amount of high frequency information transferred from the source images into the SR image. It is defined as follows: Q n, m and Q n, m are the edge strength and orientation preservation values respectively; n, m represent the image location and N, M are the size of the images respectively. Q BF (n, m) is similar to Q AF (n, m). W A (n, m) and W B (n, m) reflect the importance of Q AF (n, m) and Q BF (n, m), respectively. The dynamic range of Q AB/F is [0 1] and it should be as close to 1 as possible.
The metric Q 0 evaluates the degree of distortion of the output image. It combines three factors of image distortion; loss of correlation, luminance distortion and contrast distortion. It is calculated between the source image A and the SR image F as follows: where, σ af represents the covariance between A and F. σ a , σ f denote the standard deviations of A and F. a, f represent the mean values of A and F, respectively. Then: Note that -1 ≤ Q 0 ≤ 1 and it should be also as close to 1 as possible.
The metric Q W further takes the salience of information into account. It is calculated among the images A, B and F as follows: where, (w) represents the relative salience of A compared to B in the same window w and C (w) denotes the normalized salience of the window w. The metric Q E contains visual information and edge information, simultaneously. It is defined as follows: where, A`, B`, F` are the corresponding edge images of A, B, F, respectively. α is a constant that reflects the contribution if the edge images compared to the original image (it is equal = 1). The metrics Q 0 , Q W and Q E integrate the characteristics of the human visual system. The larger their values mean better results.
In order to validate the proposed methodology, its performance was further quantitatively evaluated with some implemented commonly used SR methods in terms of all the previous quantitative measures. The implemented SR methods are; Nonparametric Bayesian INLA Approximation (Camponez et al., 2012). Projection onto Convex Sets (POCS) (Panda et al., 2011), Iterative-Interpolation (Bannore, 2009), Robust (Zomet et al., 2001) and iterated back projection (Irani and Peleg, 1991). Their results are shown in Fig.  10 by zooming in subsections. All the quantitative measures are calculated between the up-sampled (using cubic convolution technique) LR bands and the implemented SR methods to assess the amount of the information transferred, as shown in Table 4. Comparing the results, it is obvious that the proposed methodology has better values than the other methods (3 out of 5 measures are higher). POCS and Iterative-Interpolation methods achieve slightly higher values in case of Q E and I FA , respectively. This indicates that the reconstructed SR image is strongly correlated with the source bands, confirms that the proposed methodology performs well in reconstructing the SR image and proves that the objective evaluation results coincide with the visual effect significantly.

CONCLUSION
Accurate registration of images is the most important and challenging aspect of multi-frame image Super-Resolution (SR). In this study, we present a proposed methodology that transforms the inconsistent sub-pixel shift of some Egyptsat-1 images to a reliable one to reconstruct a SR image. We demonstrate how this efficient methodology provides substantial accuracy. A SR reconstruction method based on the multi-resolution wavelet decomposition fusion was used. A core of the steerable wavelet decomposition algorithm and the normalized convolution technique has been conducted. We aimed at recognition of objects with sizes approaching its limiting spatial resolution scale. We were also interested in an objective evaluation of the proposed methodology. Therefore, its performance was compared, in terms of different quantitative measures, with some implemented commonly used SR methods to verify its usefulness and effectiveness. The implemented methods are; Nonparametric Bayesian, POCS, Iterative-Interpolation, Robust and Iterated Back Projection. The results have shown that the proposed methodology yield better values compared to most of the other implemented ones. The results have also revealed that POCS and Iterative-Interpolation methods achieve slightly higher values in case of QE and IFA, respectively. Therefore, this research demonstrates that the proposed methodology provides significant improvements in the SR results.