NOVEL COLOR FILTER ARRAY DEMOSAICING IN FREQUENCY DOMAIN WITH SPATIAL REFINEMENT

The main idea behind wavelet based demosaicing with spatial refinement is to reconstruct the full resolution color image from the mosaiced image. In this study, a new effective wavelet based demosaicing algorithm for interpolating the missing color components in Bayer’s Color Filter Array (CFA) pattern is proposed. This interpolation technique uses the interchannel correlation among the high frequency subbands to determine the missing pixels in each color channel, followed by a refining step in spatial domain which uses non-iterative technique that enforces color difference rule with fewer computations. As a result, the proposed demosaicing method yields better performance than bilinear, edge based and subband based demosaicing methods.


INTRODUCTION
Digital cameras use a Charge Coupled Device (CCD) to capture the color signal of objects. The color filter array is placed between the lens and the sensors in the charge coupled device. CFA has one color filter element for each sensor and it manages one color sample at a time. The CFA is used to capture all three color channels at the same time and reducing the complexity and cost of the digital cameras. The Bayer pattern shown in Fig. 1 is widely used CFA pattern (Bayer, 1976) in which, the sampling of Green (G) color component is twice as compared to Red (R) and Blue (B) color components. At each pixel only one color component is present, the missing two colors have to be interpolated from the existing pixels. The process of reconstructing a full resolution color image from Bayer sample is known as CFA demosaicing. The improper demosaicing method leads to visual artifacts that are around edges and color misregistration artifacts that degrade the color resolution.
The different approaches have been applied for demosaicing algorithm are as follows: The first category is the simple non adaptive algorithms such as nearest neighborhood (Jean, 2010), bilinear interpolation (Blu, 2004), that is used to interpolate the missing pixel values. The bilinear interpolation is the simplest color interpolation method (Kim et al., 2006;Mohanbabu and Renuga, 2012). In bilinear interpolation, the missing color components are calculated by taking average of neighbouring pixels to determine the missing green, red and blue color of each pixel (Jean, 2010;David et al., 2002). It blurs the edge region of the resulting image because of the absence of object boundaries and produces the highly visible artifacts.
The second category demosaicing algorithm uses interchannel and intra channel correlation (Nakanishi et al., 1998;Cok, 1987; to interpolate the missing color components. Intra channel correlation is based on the fact that the color values of the neighbouring pixels have small variations. The

Science Publications
JCS missing pixels can be interpolated with the use of neighbouring pixels. Inter channel correlation (Cok, 1987;Pei and Tam, 2003;Weldy, 1988;Adams, 1995) uses color ratio rule (Kimmel, 1999) and color difference rule (Pei and Tam, 2003;Su, 2006) which exploit the correlation among color planes. Color difference rule is mainly used in many algorithms (Pei and Tam, 2003;Su, 2006;Kim et al., 2010;Chen et al., 2012) due to its simplicity. The color difference rule states that the difference between two color channels (R, G, B and G) in every pixel location within a boundary is nearly a constant value.
The third category is edge-directed interpolation which is an adaptive approach that detects spatial features present in the neighborhood pixel (Newlin and Monie, 2013;Baharav and Kakarala, 2002;Laroche and Prescott, 1994;Hibbard, 1995;Adams and Hamilton Jr, 1996;Li and Orchard, 2001;Kim et al., 2010;Chen et al., 2012). The direction of edge is estimated by the horizontal and vertical gradients (Hwang and Lee, 2004) from the color images. The missing color components can be calculated along the edges of the image. The resulting demosaiced images are sharper with less blurring artifacts (Chen et al., 2008). This algorithm provides the good quality on the edge region but it yields poor result in problematic regions of the image.
The fourth category explains the subband interpolation (Lin and Su, 2007;Chen et al., 2008;Driesen and Scheunders, 2004). In frequency domain, the color planes are divided into series of subbands with different frequency information. The low frequency band gives the coarse information about the images. High pass bands reveal the fine information in images which corresponds to edges. There exists a strong correlation between the high frequency bands of different color planes (Chen et al., 2008). For interpolation, the demosaicing algorithms (Altunbasak et al., 2002;Chen et al., 2008;Su and Kao, 2009) mainly uses highly correlated high frequency bands. The demosaicing algorithms (Altunbasak et al., 2002;Li, 2005;Chen et al., 2008) are iterative, that introduces high computations.
In Wavelet-based color filter array demosaicing (Chen et al., 2008) starts with the bilinear interpolation which acts as an initial interpolation technique. This initial interpolation algorithm uses the intra channel interpolation. Down-sampled images are formed from the initially interpolated image. The high frequency subbands of missing down-sampled images are interpolated with the help of high frequency components of other known color planes. Finally an iterative subband method is used for artifact reduction.
Although subband synthesis algorithm produces better results than previous algorithms, the algorithm needs too many computations due to its iterative artifact reduction method. The proposed method replaces this iterative method by a non-iterative technique which exploits inter channel correlation in spatial domain. The initial estimation with bilinear interpolation is fine tuned using adaptive edge based color plane interpolation (Kim et al., 2006;Chen et al., 2012). The proposed algorithm uses both spatial and frequency domain techniques during interpolation.
This study is organized as follows, section II explains the proposed method and each sub section explains each part of the system. Section III briefs the steps involved in the proposed methodology. Section IV reports the experimental results. CPSNR and PSNR are used as comparative measure. Here 24 Kodak images database are used for comparison between various demosaicing methods with the proposed algorithm.

WAVELET BASED DEMOSAICING WITH SPATIAL REFINEMENT
The demosaicing algorithm starts with bilinear interpolation technique to interpolate missing R, G, B colors. Although these algorithms are computationally efficient, but due to their fixed pattern of interpolation without considering edges of object, creates artifacts. Since edges play a main role in images, interpolation of channel is done by considering the direction of edges. Then the interpolated image is subdivided into subimages and 2D discrete wavelet transform is applied to separate each color channel into Low-Low (LL), Low-High (LH), High-Low (HL) and High-High (HH) subbands. Then the coefficients of the wavelet transform are modified by making use of inter channel correlation. After updation, inverse discrete wavelet transform is applied to bring back to spatial domain and post processing is done using color difference rule. The algorithm flow is shown in Fig. 2.

Bilinear Interpolation
Bilinear interpolation is an upsampling method that uses the distance-weighted average of the four nearest pixel values to estimate a missing pixel value. This interpolation method takes pixels in the Bayer pattern and ignores the information about the edge regions. Bilinear algorithm interpolates every missing pixels in a fixed pattern. The missing pixel values are calculated by taking average value of the four adjacent pixels in each color plane. This introduces large errors that lead to blur the interpolated image.

JCS
The green, red and blue color components are interpolated using the equations given in Demosaicing with the Bayer Pattern (Jean, 2010).

Edge Adaptive Interpolation
Edge adaptive method is an adaptive approach that is used to interpolate the missing pixel values along the edge patterns of the color channels. In this method, the edge directions such as horizontal and vertical directions are calculated according to the edge region (Pei and Tam, 2003;Chang and Chen, 2012;Gunturk et al., 2005). The edge directions are defined by the horizontal and vertical gradients to detect the pixel location. This interpolation is used to avoid choosing the direction across edges. The horizontal and vertical gradients are compared, if the horizontal gradient is greater than the interpolation is along vertical direction. If the vertical gradient is greater, then the green channel is estimate along the horizontal direction. If the horizontal and vertical gradients are equal then the interpolation is calculated by taking the average of its four neighbouring pixels. The missing green pixel location can be obtained from the neighbouring green pixels (Kim et al., 2006).
The missing green pixel G (4, 3) in Fig. 1 at blue row can be estimated by using Equation 1 The missing green pixel G(3,4) at Red row can be estimated using Equation 3 and 4 as follows: ∆H and ∆V are horizontal and vertical gradient functions used to find edge directions.

Wavelet Demosaicing
The wavelet based demosaicing method is based on the fact that the high frequency subbands {LH, HL, HH} of different color planes are highly correlated (Chen et al., 2008) and have constant variation between the color planes which is proved by using Mean Absolute Error (MAE) metric in (Chen et al., 2008). This property can be used to update the high frequency subband coefficients of the interpolated color pixels with the help of high frequency subbands of known color pixels.
In Fig. 3 the pixels with '*'are the color pixels obtained from Bayer's pattern. All other pixels are missing pixels in Bayer's pattern found out by using initial interpolation (Bilinear interpolation and edge adaptive interpolation).  The interpolated color image is separated into subimages according to the positions in Bayer's array {00, 01, 10, 11} for each color channel. The subimages are formed by collecting all pixels having same subscript and a total of 12 subimages are formed (r 00 , r 01 , r 10 , r 11 , g 00 , g 01 , g 10 , g 11 , b 00 , b 01 , b 10 , b 11 ). Each subimage is divided into four frequency bands {LL, LH, HL, HH} using Two Dimensional Discrete Wavelet Transform (2D-DWT). The high frequency subbands {LH, HL, HH} of interpolated pixels are updated from observed pixels (g 00 , g 11 , r 01 , b 10 ) in CFA.

Refining R and B at "00" place
In '00' positions g 00 is the known color. The missing other two colors R and B have to be refined with the help of g 00 :

Spatial Refinement
In subband synthesis demosaicing (Chen et al., 2008) the refining of demosaiced color planes are done in frequency domain using wavelet transform and the refining procedure was iterative. This needs many computations. These complexities are solved by using a spatial non iterative technique that reduces the complexity.
During the refinement step red and blue channels are alone updated leaving green channel unaltered. Since in Bayer's pattern 50% of the pixels are green, 25% of the pixels are red and 25% of the pixels are blue more number of red and blue pixels are interpolated pixels compared to green pixels. So red and blue pixels are more prone to errors. The red and blue pixels are refined using color difference rule which is used as a tool to explore interchannel correlation (Li, 2005).
The expression for finding color difference signals are found by taking the difference between the red and green and blue and green channel. Due to the majority of green pixels 'G' is taken as reference: The color difference signals CD R and CD B are found in every pixel location. The Bayer pattern in Fig. 1 is taken as reference to refine the procedure. The native green pixels are considered as two categories, green pixel at the red pixel row and green pixel at blue pixel row.
The blue pixel at native green pixel (blue pixel row) is refined by: CD 2,1 CD 2,3 2 B 2,2 G 2,2 CD 2,2 The blue pixel at native green pixel (red pixel row) is refined by: Likewise blue pixels in all native green pixel locations are refined.
The blue pixels at native red pixels are refined by: CD 2,2 CD 4,2 1 CD 3,2 4 CD 3,1 CD 3,3 B 3,2 G 3,2 CD 3,2 Likewise blue pixels in all native red pixel locations are refined. The red pixels are refined in similar manner of blue pixel. The procedure used for refining uses simple expressions hence no complex processing required as compared to subband based demosaicing method (Chen et al., 2008).
The performance of the proposed work is measured in terms of CPSNR for whole image and PSNR for each channel. These performance measures are comparative which takes original input image as reference which reflects how far the reconstructed image resembles like original input image (Kim et al., 2006;Jean, 2010;Tian et al., 2012;Fan et al., 2013). The CPSNR and PSNR measurement can be defined as follows Equation 13 and 14: Where: H,W = The height and width of image, I 0 k (i,j) = The original image, I d k (i,j) = The reconstructed image using demosaicing algorithm K = The corresponding color plane (R,G,B).
The steps involved in Wavelet based demosaicing with spatial refinement are as follows.

Initial Interpolation
The missing pixels in Bayer pattern are interpolated using bilinear interpolation algorithm (Jean, 2010) which uses intra channel correlation for interpolation.

Edge Based Interpolation
Since edges play a main role in enhancing the image. Inorder to reduce the errors during interpolation, interpolation is done along edge directions by using edge adaptive interpolation using the Equation 1-4.
The high frequency components of each sub image is updated by using Equation (5-8).
Inverse 2D-DWT is applied to synthesis the subimages.

Refining Step
Non Iterative algorithm which makes use of interchannel interpolation is used to refine the Red and Blue color values as per Equation (9-12).

RESULTS
In this study, the wavelet based demosaicing with spatial refinement technique is compared with the previous interpolation methods such as bilinear interpolation, edge based interpolation and subband synthesis demosaicing algorithms. The performance measures are estimated by using 24 kodak images shown in Fig. 4 with size 768×512 pixels.
The CPSNR values are tabulated in Table 1. It is clear that, the proposed method gives better CPSNR values than the existing demosaicing algorithms.    The CPSNR values are calculated by performing the MATLAB source code. The average CPSNR of the proposed demosaicing method is increased than the other three interpolation methods. The Table 2 shows the PSNR values for every color channels. From the results, it can be observed that for edge based interpolation, G channel PSNR is improved compared to bilinear method. Subband synthesis method shows better performance than other two algorithms due to processing of high pass bands and a wavelet based post processing technique. The proposed method has better PSNR values for every color channel compared to other demosaicing algorithms.

CONCLUSION
In this study, an effective demosaicing algorithm has proposed, which uses both spatial and frequency technique for interpolation. The edge adaptive color plane interpolation is used to determine the edge direction of the missing pixel by using the neighbouring color components directional information. Wavelet based demosaicing method is used to estimate the missing pixels by interchannel correlation among high frequency subbands. This enhances the fine details in the image and reduces the color artifacts with less computation. The spatial refinement technique enforces the color difference rule to refine the missing color components. The experimental result shows that the proposed demosaicing technique yields better performance than other existing interpolation methods such as bilinear, edge based and subband based interpolation methods and produces the better quality image.