EXPLOITATION OF INTER-COLOR CORRELATION FOR COLOR IMAGE DEMOSAICKING Sunil Prasad Jaiswal, Oscar C. Au, Vinit Jakhetiya, Yuan Yuan, Haiyan Yang Department of Electronics and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong ABSTRACT Image demosaicking or color filter array interpolation is a process of interpolating missing color samples to reconstruct a full color image. In general, existing algorithms assume that the high frequency components such as edges, texture etc. of different color channels are similar and thus take an advantage of it to estimate the missing samples. In this paper, we efficiently analyze the relationships of intra and intercolor correlation among the channels and observe that such assumption fails in most cases. In view of this observation, we propose a scheme that exploits the correlation between different color channels much effectively than the existing algorithms. Experimental results demonstrate that the proposed algorithm outperforms the existing methods both in terms of Peak Signal to Noise Ratio (PSNR) and visual perception. Index Terms Bayer color filter array, color difference interpolation, high frequency components, demosaicking. 1. INTRODUCTION Digital Color Images are usually represented by three color values at each pixel. To completely measure the image, it is desirable to have three charge-coupled devices (CCD) to capture red (R), green (G), and blue (B) components which is a costly approach. Thus to reduce the production cost many cameras use a single sensor covered with a color filter array (CFA). The CFA allows only one color to be measured at each pixel. The most popular design used to achieve this is the Bayer pattern shown in Fig. 1. Thus two missing color elements at each pixel must be estimated from the adjacent pixels. This process is called CFA interpolation or demosaicking. Several demosaicking methods have been proposed in literature in recent years [1,2]. The simplest way to do demosaicking is to apply low pass filter (LPF) independently to each channel which introduces color artifacts and blur the edges as well. The reasons why the color correlation is used based on the observation that all color channels have similar edges or texture structure [3] and thus assume that color difference (R-G or B-G) are low pass signals. Since sampling frequency of G is higher than that of R and B channels (shown in Fig. 1), thus most of Bayer demosaicking algorithms first interpolate the green (G) channel and then use it to drive the interpolation of the red and blue channels [3]-[10]. Zhang et al. [4] estimate the green samples using Linear Minimum Mean square estimate (LMMSE) based filter technology and then the other two color channels are reconstructed based on the color difference assumption. Paliye et al. [5] extended the method in [4] by adopting a new filtering strategy, the LPA-ICI filter [5]. Tang et al. [6] proposed to divide the image into sub image and merge them in frequency domain with linear combination. However, method in [7] & [8] adopted non-local characteristics based features for interpolation which increase the accuracy of the predictor. Methods in [9] interpolated the residual image instead of color difference image assuming that the residual image are more flatter. Most of the above mentioned algorithms are based on color difference interpolation (CDI) and assume that color difference signals are low pass signals. We efficiently analyze the given assumption and observe that it fails in most cases which produces false edges and thus reduces the overall quality of the image. To solve this issue, we propose a novel algorithm which exploits the inter-color correlation property of the channels and thus achieve the superior performance. The paper is organized as follows. In Section 2, Color Difference Interpolation (CDI) method is explained. Section 3 includes distortion analysis of CDI method and proposed demosaicking algorithm. Simulation results and concluding remarks are given in section 4 and in section 5 respectively. Fig. 1. Bayer pattern (US Patent 3971065). 2. COLOR DIFFERENCE INTERPOLATION (CDI) Most of the demosaicking methods make an assumption that the high frequencies between R, G and B are largely identical. We consider only R and G component in discussion 978-1-4799-5751-4/14/$31.00 2014 IEEE 1812
as B may be treated in a similar manner. Interpolated green component is denoted as Ĝ in our discussion. First interpolate Green component since G pixels are sampled twice than R or B pixel and thus can preserve high frequencies of G channel. Then, color difference K R = R a Ĝa is estimated at the available R pixels, where R a and Ĝa denote the R and Ĝ channels sampled at the available R positions as shown in Fig.1. The reason behind color difference phenomenon can be interpreted as each color component such as R (or G) can be decomposed into a low-frequency term R l (or G l ) and a high frequency term R h (or G h ). Since the high frequency components of different colors tend to be similar [3], then R G = (R l + R h ) (G l + G h ) = (R l G l ) (1) Thus color difference doesn t contain high frequency component. Therefore if interpolated Ĝ is fully available by some interpolation process, then interpolated red component ( ˆR CDI ) by CDI method can be recovered from (2) ˆR CDI = ζ{k R } + Ĝ = ζ{r a Ĝa} + Ĝ (2) where ζ{.} denotes low pass filtering i.e, low pass filter (LPF) is applied to the color difference image (K R ) and then added to the interpolated Ĝ image to get the interpolated R component. Assuming the low pass filtering of R a and Ĝa gives exactly the low frequencies of R and Ĝ respectively, the interpolated ˆR CDI from (2) can be written as: ˆR CDI = ζ{r a Ĝa} + Ĝ = R l Ĝl + Ĝ (3) In a similar manner, blue component can be estimated. 3. PROPOSED ALGORITHM We observe that the assumption CDI make in (1) fails in some cases and hence produces false edges which reduces the overall quality of the image. To solve this issue, we make an analysis of distortion obtained by a simple low pass filter and by CDI method and compare the same. 3.1. Distortion Analysis of Low pass filter (LPF) The simplest way to demosaicking is to apply low pass filter to each channel independently. Using the same LPF (as used in (2)), the interpolated red component ( ˆR LP F ) is given by ˆR LP F = ζ{r a } = R l (4) The distortion between original R component and interpolated ˆR LP F component can be estimated as: D LP F = R ˆR LP F = R R l = R h (5) where R h is the high pass filter output of R component. Assuming D LP F and R h follows Laplace Distribution [11] with zero mean i.e E[D LP F ] = 0 and E[R h ] = 0, variance of final distortion (σ 2 D LP F ) is equal to variance of R h (σ 2 R h ) i.e, σ 2 D LP F = E[(R R l ) 2 ] = E[R 2 h] = σ 2 R h (6) 3.2. Distortion Analysis of CDI Method As discussed in previous section, the interpolated ˆR CDI from (3) can be written as: ˆR CDI = R l Ĝl + Ĝ = R l + Ĝh (7) where Ĝh (Ĝ Ĝl) is the high pass filtered output of Ĝ component. The distortion between original R component and interpolated ˆR CDI component can be estimated as: D CDI = R ˆR CDI = R R l Ĝh (8) Incorporating (5) into (8), we get D CDI = (R h ˆ G h ), D 2 CDI = R 2 h +Ĝ2 h 2 R h Ĝh (9) Assuming Ĝh and D CDI follows Laplace Distribution [11] with zero mean, variance can be given by σ 2 = Ĝ E[Ĝ2 h h ] and σ 2ˆDCDI = E[ ˆD CDI 2 ] respectively. Average distortion is calculated by taking expected value on both sides of (9) i.e, E[D 2 CDI] = E[R 2 h] + E[Ĝ2 h] 2 E[R h Ĝh] (10) Correlation coefficient estimated between Ĝh and R h is given by, ρ Rh Ĝ h = E[R h Ĝh]/(σ Rh σĝh ). Using (6) and (10), we get σ 2 D CDI = σ 2 D LP F + σ 2 Ĝ h 2 σĝh σ Rh ρ Rh Ĝ h (11) Now, above equation is the variance of distortion of CDI method for R component. In (11), ρ Rh Ĝ h ( 1, +1) and it tells about the similarity between high frequency component of two channel. Higher value of ρ Rh Ĝ h means edge structure in R and Ĝ component are quite similar and lesser ρ R h Ĝ h means edges are not similar. We can write (11) as: σ 2 D CDI σ 2 D LP F = σĝh (σĝh 2 σ Rh ρ Rh Ĝ h ) (12) For CDI method to work better than simple LPF in terms of distortion i.e, σd 2 CDI < σd 2 LP F, from (12) we can write σĝh 2 σ Rh ρ Rh Ĝ h < 0, which reduces to ρ Rh Ĝ h > 0.5 σĝh /σ Rh (13) In terms of distortion, we can conclude that CDI method can work better than LPF only if the condition in (13) is satisfied. 3.3. Proposed Demosaicking Scheme From the above analysis, we observe that CDI method does not work in some cases and thus simple LPF should be used to increase the prediction accuracy for demosaicking. In view of this observation, we propose a linear combination of LPF method and CDI method on a block by block basis to generate a more accurate prediction for demosaicking and optimal weights are estimated in LMMSE sense for each block. We 978-1-4799-5751-4/14/$31.00 2014 IEEE 1813
denoted the proposed interpolated red component by ˆR and it is expressed as: ˆr = w 1ˆr LP F + w 2ˆr CDI (14) Here ˆr ˆR is to be interpolated block of size M N and ˆr LP F ˆR LP F, ˆr CDI ˆR CDI are reconstructed blocks of same size (M N) obtained by LPF method and CDI method respectively. w 1 and w 2 are the weighted coefficient for the combinations. Thus the distortion (d) between original block (r R) and interpolated block (ˆr) can be written as: d = r ˆr = r (w 1ˆr LP F +w 2ˆr CDI ) = w 1 d LP F +w 2 d CDI (15) Here, d LP F D LP F and d CDI D CDI are the distorted blocks obtained by method LPF and CDI respectively (D LP F, D CDI is defined in (5) and (8)). To get the optimal weights (w 1, w 2 ) in LMMSE sense, the problem can be formulated as below min E[d 2 ] w 1,w 2 s.t w i = 1 i=1,2 (16) To minimize the residue energy (E[d 2 ]) of each block, differentiate it with respect to w 1 and w 2 and the optimal weights can be expressed as: { w 1 = E[d CDI (d CDI d LP F )]/E[(d CDI d LP F ) 2 ], w 2 = E[d LP F (d LP F d CDI )]/E[(d CDI d LP F ) 2 ] (17) So by substituting (17) in (14), we can efficiently interpolate the missing samples of the block (ˆr) for red component. Unfortunately, to estimate d LP F and d CDI in (17) we need original block (r) which is not available in practice. In our experiments, we propose to use the resultant image of the method [9] as an estimation of original block (r). 3.4. Algorithm steps The main steps of the algorithm are as follows: 1. Interpolate the green component (Ĝ) first by some interpolation process. In our experiment we propose to use method [9] to get the Ĝ component. 2. To interpolate the missing samples of a block (ˆr) in red component, its corresponding ˆr CDI and ˆr LP F are computed. 3. Due to unavailability of original block (r), we use the resultant image (`r) obtained by the method [9] as an estimation of original block (r). Thus the weighted parameters (w 1, w 2 ) can be calculated from (17). Using the weighted parameters, ˆr can be estimated from (14). 4. Repeat step 2 and step 3 to reconstruct the whole image. In a similar manner, blue component can be reconstructed. 4. SIMULATION RESULT We implemented the proposed algorithm and compared its performance with existing methods. For this purpose, we used standard color image datasets (the IMAX dataset) given in [8]. It consists of 18 images and the image size is 500 500. In our simulation, we select ten random images from the dataset and is shown in Fig. 2. For better evaluation, we have shown both objective as well as subjective quality and compared the same with various methods. We compared the proposed algorithm with directional LMMSE [4], local polynomial approximation [5], self similarity [7], local directional interpolation [8] and residual based interpolation [9] methods. Results with objective quality is shown in Table 1 and Table 2 while subjective quality is shown in Fig. 3. Our algorithm Fig. 2. Original Test Images used in the experiment focus on analyzing the inter-color correlation nature of CDI method for interpolating R and B component. Thus in our simulation, we use method [9] to get the Ĝ component. The performance of the reconstruction of red and blue component is dependent on the accuracy of the Ĝ component. Better the interpolation accuracy of Ĝ, better will be the reconstruction accuracy of red and blue component. In Table 1, PSNR comparison for red component is given. It can be found that the proposed algorithm has better PSNR than existing methods in most of the test images. On an average, proposed algorithm exceeds method [8] and method [9] in PSNR by 0.52 db, 1.27 db respectively. In Table 2, PSNR comparison for blue component is given and it can be observed that on an average, proposed algorithm exceeds method [8] and method [9] in PSNR by 0.91 db, 0.20 db respectively. To show the effectiveness of proposed algorithm, visual comparison of a portion of Fig 2 (e) is shown. From visual comparison (in Fig. 3), we can say that proposed algorithm can sharply interpolate the image and leads to clearer and sharper reconstructed image without color artifacts. Our Algorithm combines LPF method and CDI method on a block by block basis and empirically we kept the block size as 6 6 in our simulation. The block size can vary for higher resolution of images. We use simple averaging filter as low pass filter (used in equation (2) and (4)) in our simulations. Better low pass filter can be used to improve the performance. 978-1-4799-5751-4/14/$31.00 2014 IEEE 1814
Fig. 3. (a) Portion of original image (Fig. 2(e)) (b) Method [4] (c) Method [5] (d) Method [6] (e) Method [9] (f) Proposed Algorithm. Table 1. PSNR (db) Comparison for red component. Pro refers to proposed algorithm and Avg refers to average. Img [4] [5] [7] [8] [9] Pro (1) 26.94 27.44 27.28 29.29 29.16 29.30 (2) 32.60 32.18 32.81 33.05 34.04 34.15 (3) 30.38 32.43 31.10 35.05 36.80 37.02 (4) 37.20 35.22 36.61 36.09 37.09 37.42 (5) 35.45 35.64 35.31 36.31 34.28 37.40 (6) 32.39 33.50 33.72 35.49 33.24 35.67 (7) 34.70 35.16 36.33 38.26 36.72 37.73 (8) 36.91 37.12 38.16 39.84 37.87 39.02 (9) 35.32 35.85 36.23 36.95 35.78 37.24 (10) 28.32 30.10 29.24 32.14 30.54 32.20 (Avg) 33.02 33.45 33.67 35.24 34.55 35.72 Table 2. PSNR (db) Comparison for blue component. Pro refers to proposed algorithm and Avg refers to average. IMG [4] [5] [7] [8] [9] Pro (1) 25.12 25.29 25.12 26.71 26.98 27.23 (2) 31.73 29.99 30.93 30.31 31.73 31.16 (3) 30.72 30.62 29.48 31.16 30.72 31.40 (4) 37.29 34.10 36.38 34.49 35.62 35.66 (5) 36.99 35.45 36.76 36.67 38.06 38.20 (6) 34.66 33.36 35.38 36.30 36.48 36.36 (7) 35.55 35.49 36.13 36.83 37.34 37.50 (8) 35.75 33.79 36.81 37.66 39.37 39.52 (9) 37.30 36.71 37.90 38.99 39.18 39.00 (10) 27.77 28.88 28.38 30.91 31.77 32.10 (Avg) 33.28 32.26 33.32 34.01 34.72 34.82 5. CONCLUSION In this paper, we proposed a novel color demosaicking algorithm based on inter-color correlation among the channels. We made a detailed analysis of inter-color correlation among the channels and compared the distortion obtained by simple LPF and CDI method. In view of this analysis, we proposed an approach that exploits the correlation between different color component and neighborhood pixels much effectively than existing algorithms. Experimental results demonstrate that proposed algorithm outperforms the existing methods both in terms of subjective and objective quality. 6. ACKNOWLEDGMENT This research was supported by the Hong Kong Government Innovation and Technology Fund and the State Key Laboratory on Advanced Displays and Optoelectronics Technologies (Project No: ITC-PSKL12EG02). 7. REFERENCES [1] B.Gunturk, J. Glotzbach, Y. Altunbasak, R.schafer, and R. Mersereau, Demosaicing: Color filter array interpolation,. in IEEE Signal Processing Magazine, vol. 22, no. 1, Jan 2005. [2] X. Li, B. Gunturk, and L. Zhang, Image demosaicing: A systematic survey,. in proc. SPIE, vol. 6822, p. 68221J, 2008. [3] B. K. Gunturk, Y. Altunbasak, and R. M. Mersereau, Color plane interpolation using alternating projection,. in IEEE Trans. On Image Processing, Vol. 11, No. 9, 2002. 978-1-4799-5751-4/14/$31.00 2014 IEEE 1815
[4] L. Zhang and X. Wu, Color demosaicking via directional linear minimum mean square-error estimation, in IEEE Trans. On Image Processing, Vol. 14, No. 12, 2005. [5] D. Paliy, V. Katkovnik, R. Bilcu, S. Alenius, and K. Egiazarain, Spatially adaptive color filter array interpoaltion for noiseless and noisy data, in Int. Journal of Imaging Systems and Technology, vol. 17, no. 3, pp 105-122, 2007. [6] W. Tang, O.C. Au; X. Wen; Y. Yang; L. Fang LMMSE frequency merging for demosaicking, in ICME 2009. IEEE International Conference on, pp.229,232, June 2009. [7] A. Buades, B. Coll, J. -M. Morel, and C. Sbert, Selfsimilarity driven color demosaicking, in IEEE Trans. Image Processing, Vol. 18, No. 6, 2009. [8] L. Zhang, X. Wu, A. Buades, and X. Li, Color demosaicking by local directional interpolation and nonlocal adaptive thresholding, in Journal of Electronic imaging, Vol. 20, No. 2, 2011. [9] D. Kiku, Y. Monno, M. Tanaka and M. Okutomi, Residual interpolation for color image demosaicking, in IEEE Int. Conf. on Image Processing (ICIP), Sept., 2013. [10] J. S. Ho, O.C. Au, J. Zhou, Y. Guo, Inter-channel demosaicking traces for digital image forensics, in ICME 2010. IEEE International Conference on, pp. 1475-1480, July 2010. [11] J. Zhou, O.C. Au, Determining the Capacity Parameters in PEE-based Reversible Image Watermarking, in IEEE Signal Processing Letter, Vol. 19, No. 5, pp. 287-290, May 2012. [12] J. Dai, O.C. Au, C. Pang, F. Zou Color Video Denoising Based on Combined Inter-Frame and Inter-Color Prediction, in Circuits and Systems for Video Technology, IEEE Transactions on, vol.23, No. 1, pp. 128-141, Jan. 2013. [13] X. Zhang, M.T. Sun, O.C. Au Joint Denoising and Demosaicking of Noisy CFA Images based on Inter-color Correlation, in Proc. of IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 4-9 May. 2014, Florence, Italy. 978-1-4799-5751-4/14/$31.00 2014 IEEE 1816