FAST REGISTRATION METHODS FOR SUPER-RESOLUTION IMAGING. Jari Hannuksela, Jarno Väyrynen, Janne Heikkilä and Pekka Sangi

Transcription

1 FAST REGISTRATION METHODS FOR SUPER-RESOLUTION IMAGING Jari Hannuksela, Jarno Väyrynen, Janne Heikkilä and Pekka Sangi Machine Vision Group, Infotech Oulu Department of Electrical and Information Engineering P.O. Box 4500, FIN University of Oulu, Finland ABSTRACT Super-resolution image reconstruction can be viewed as a two stage process: image registration and fusion. The current solutions to perform processing generally need too much computational power to be applied in mobile devices with limited resources. In this paper, two efficient registration methods are introduced for these platforms. The proposed schemes are the block matching using uncertainty analysis and the phase correlation with block processing. The methods are broadly compared to other well-known approaches presented in the literature. Experiments with synthetic and real image sequences clearly demonstrate the performance of the methods and indicate that proposed methods can be applied in practise. 1. INTRODUCTION The increasing availability of mobile phones equipped with a camera provide an opportunity to more versatile and portable imaging devices and applications. Often, the image quality achievable is restricted due to limited optics and imaging resolution. On the other hand, the computational power of mobile devices is improving all the time and it is possible to apply proper super-resolution image reconstruction algorithms to achieve enhanced images using the same platform. Super-resolution image enhancement utilizes several low resolution images to reconstruct a single high resolution image. A good overview of existing work is given by Park et al. [1]. Two main stages can be separated for the reconstruction process: image registration and fusion. These stages can be implemented separately or simultaneously. In order to register images, the motion parameters of slightly differently positioned low-resolution images are estimated, and then images are warped into the same coordinate system. In the fusion stage, the information of low-resolution frames is extracted and used for superresolution image reconstruction. Registration in subpixel accuracy is a fundamental requirement for success in the reconstruction process. The registration can be done in the spatial or in the frequency domain. The super-resolution problem was first addressed by Tsai and Huang [2]. They presented a frequency domain method considering only global translational motion between frames and linear space invariant blur in the imaging process. Also, many other frequency domain registration methods assume that two images differ in phase shift only, that can be measured from their phase correlation. The phase correlation can be extended to subpixel precision using a technique described by Foroosh et al. [3]. Recently, Vandewalle et al. [4] proposed a frequency domain approach to estimate a horizontal and a vertical shift, and also a planar rotation. However, as a fusion method it utilizes only a rather simple bicubic interpolation. Typically, frequency domain methods are limited to translational motion, but in many super-resolution applications also some rotation and scale changes are present. Due to obvious limitations of the frequency domain approaches most of the later work concentrates on spatial domain methods. Irani and Peleg [5] proposed an iterative back-projection approach inspired by computer aided tomography. Their method estimates a high-resolution image by iteratively back projecting the error between simulated and observed low resolution images. The imaging model used includes both optical blur and spatial quantization. It has been shown to produce quite good results and tolerate noise well. Zomet [6] improved the method by taking the median of the errors in the different backprojected images. This should be more robust in the case of outliers. Farsiu et al. [7] proposed a new robust algorithm to produce sharper high-resolution images that uses L 1 norm instead of more common L 2 minimization. Typically, in the super-resolution reconstruction, the blurring process is assumed to be known. However, in practise this process is usually partially or fully unknown. Therefore it is often necessary to incorporate the blur identification into reconstruction process. Šroubek and Flusser have presented a super-resolution method utilizing multichannel blind deconvolution [8] for this purpose. However, their approach can handle only translational

2 motion between frames. A number of work has been presented for superresolution image enhancement. However, real-time implementation issues are only seldom addressed. In this paper, we present two efficient image registration solutions that are suitable for platforms where computing power is limited. The proposed schemes are the block matching using uncertainty analysis and the phase correlation with block processing. The former operates in the spatial domain and the latter in the frequency domain. Moreover, these methods can possibly take advantage of hardware accelerators designed for video coding systems. The methods are used together with Irani and Peleg [5] fusion algorithm in order to evaluate their feasibility. The experiments are carried out with synthetic and real image sequences and compared to some state of the art super-resolution methods. 2. PROPOSED REGISTRATION METHODS The accurate registration of images that is also referred as motion estimation is needed to model the transformation between low-resolution frames. We utilize the four-parameter Euclidean similarity model for approximating geometric relationship between frames. It can present 2-D motion consisting of translation, rotation, and scaling. The displacement v of a feature located at p = [x, y] T can be represented using v = v(θ, p) = [ 1 0 x y 0 1 y x ] θ (1) where θ = [θ 1, θ 2, θ 3, θ 4 ] T is a vector of model parameters, θ 1 and θ 2 are related to translational motion in x- and y- directions respectively, and θ 3 and θ 4 contain information about 2-D rotation φ and scaling s: θ 3 = s cos φ 1 (2) θ 4 = s sin φ. (3) We propose two alternative approaches to estimate θ for the apparent dominant motion between images. First, a spatial domain method based on block matching with uncertainty analysis is presented [9]. Secondly, a frequency domain method utilizing distinct block processing with a subpixel phase-correlation [3] is introduced Block matching with uncertainty analysis The method is based on estimation of local motion features, which encode information about displacements of a sparse set of image blocks between two frames. The motion features are passed to outlier analysis, which defines the inlier features used for global motion estimation. Computation of motion features begins by selection of blocks from the anchor frame I k 1 ( ). Locations of the blocks p P are obtained by dividing the image region to N rectangular subregions and selecting one block from each subregion. In this way, we have features distributed over the image. Comparison of candidate blocks is based on analysis of image gradients. In our solution, we compute eigenvalues of the normal matrix [10]. In a subregion, we try to locate a block with strong gradient values in orthogonal directions first that is both eigenvalues of the normal matrix are large. If such a block is not found, we seek a block with strong gradients in one spatial direction that is large value for one eigenvalue. In the second phase of motion feature computation, displacement of each chosen block is estimated. As a block matching measure, we use the zero mean sum of squared differences (ZSSD) criterion, which can be defined as D(d) = x B(I k (x + d) I k 1 (x) µ(d)) 2, (4) where d = [u, v] T is the displacement, B denotes a set of block pixel coordinates, I k ( ) is the target frame, and µ(d) is the average of I k (x+d) I k 1 (x) computed over x B. The ZSSD measure can tolerate changes in lighting conditions and sensor sensitivity [11], which is crucial in our application due to automatic control of image capturing parameters such as an exposure time. In our method, ZSSD is evaluated for some range of integer displacements u and v. The surface of a matching measure values called the motion profile, is used for refinement and uncertainty analysis. The original estimate for the displacement is refined to subpixel precision by fitting of quadratic functions to the criterion values in the neighborhood of that minimum. Separate fitting in horizontal (u) and vertical (v) directions is performed in our case. To evaluate the uncertainty of the local displacement estimate, analysis of the motion profile is performed in order to detect those displacements, which are possible according to the ZSSD criterion. The main principle is to perform gradient-based thresholding for the motion profile. The set of good matches is defined as V = {d D(d) T } with the threshold T = D(d m ) + k 1 G + k 2, (5) where d m is a displacement, which minimizes the block difference measure, G is a block gradient measure, and k 1 and k 2 are constants. The gradient measure G is the sum of squared differences of neighbor pixels computed over the block in horizontal and vertical directions. Statistical justifications for this principle are given in [12]. Once thresholding has been performed, we summarize the result as a covariance matrix C v, which is the second central moment of d V with constant c = 1/12 added to the diagonal values. The constant reflects the fact that thresholding was done for integer-valued displacements.

3 We also compute the first moment over V, which is denoted d v. With these notations, a single motion feature is defined as a quadruplet (p, ˆd, d v, C v ), where p denotes the location of the block centroid in the image and ˆd is the subpixel precision displacement estimate. Global motion estimation uses the local motion features and determines a parametric motion model (1) which is supported by those features. Robust fitting of parametric models is typically based on the random sample consensus [13] or some variant of it. We present an approach, where agreement on some model takes local motion uncertainty information into account. Outlier removal takes the set of motion features as an input and outputs a set of trusted motion features, which are triplets (p, ˆd, W), where W is a 2 2 weighting matrix. This information is fed to the parametric model fitting stage, where the final result is estimated using the weighted least squares method. In outlier analysis, hypotheses about global motion are generated by drawing sufficient subsets of motion features and solving the corresponding system of equations, which is based on (1). In those equations, estimates ˆd are used for displacements d. The best hypothesis is found among generated ones, and it defines the inlier features used in the final model fitting. Uncertainty information associated with the motion features is used to support outlier analysis. The basic principle would be to evaluate the Euclidean distance between the hypothesized displacement and the estimated displacement. However, we can also use the Mahalanobis distance based on C v in our analysis. This enhances utilization of partial displacement information available at edgelike regions [9] Phase correlation with block processing The phase correlation image registration first proposed by Kuglin and Hines [14] is a well known frequency domain method. The method is based on the Fourier shift theorem between two mages f 1 and f 2, which are translated versions of each other. This can be expressed as f 2 (x, y) = f 1 (x x, y y), (6) where x and y are horizontal and vertical displacements. The relationship of their Fourier transforms F 1 and F 2 is given as F 2 (u, v) = e 2πj(u x+v y) F 1 (u, v). (7) The displacement in the spatial domain is reflected as a phase difference in the frequency domain. In order to get rid of the luminance variation influence, we normalize the cross-power spectrum by its magnitude, and obtain its phase S(u, v) = F 2(u, v)f1 (u, v) F 2 (u, v)f1 (8) (u, v). By combining Equations (7) and (8) we get S(u, v) = e 2πj(u x+v y). (9) The inverse Fourier transform of (9) is δ(x x, y y), which is the Dirac delta function centered at ( x, y) corresponding to the spatial shift between images f 1 and f 2. The displacement can be found easily by detecting a highest peak of the response. We have implemented a phase correlation method, where the current frame is divided into 16 by 16 pixel blocks and phase correlation calculation is performed for each block. Fast Fourier transform (FFT) is first performed on the two corresponding blocks in successive frames. In practice, the highest peak is detected from the phase correlation image to estimate the displacement. The subpixel accuracy for the displacement estimate is then obtained using the procedure described in [3]. After, the displacements for feature blocks have been found, the global motion between frames are estimated by solving the motion model (1) with the least squares (LS) method. However, there might be some errorneous displacement measurements whose effect should be reduced. We assume that majority of measurements are correct and utilize the robust M-estimation algorithm to improve the initial LS estimate of the global motion. In practise, we apply iteratively reweighted least squares (IRLS) method for determining M-estimates for motion parameters. 3. EXPERIMENTAL RESULTS In experiments, various methods were evaluated with synthetic and real images. First, we measured the performance of the algorithms quantitatively using a sequence of synthetic images. Image sequences of 10 frames were generated from a high-resolution (HR) base image. We used four base images A,B,C and D containing different kinds of a texture. The HR image was first randomly warped using the Euclidean similarity model described in Sec. 2. The parameters for this model were chosen based on experiments using real images captured with a mobile phone. The warped images were cropped to the size of 320 by 240 pixels. Then optical blur was added with a Gaussian point spread function of 5 by 5 pixels and variance σpsf 2 = 1. These blurred frames were down-sampled by the factor two in both directions. Finally, zero-mean Gaussian noise was added with variance σ 2 = 4 to obtain the low-resolution (LR) images. In order to achieve statistical reliability, we created 10 test sets with 10 LR images for each base image. Performance measures used were based on the peak signal-to-noise ratio (PSNR) between the reference HR image and the achieved super-resolution image. The SR image was intensity normalized before calculation. Mean PSNR computed for test sequences using methods under

4 (a) (b) (c) (d) (e) (f) Fig. 1. Synthetic test A: (a) LR image (b) Vandewalle (c) BMU (d) BPC (e) Zomet (f) Farsiu (a) (b) (c) (d) (e) (f) Fig. 2. Synthetic test A: (a) LR image (b) Vandewalle (c) BMU (d) BPC (e) Zomet (f) Farsiu PSNR [db] Method A B C D Vandewalle [4] BMU + Irani [5] BPC + Irani [5] Zomet [6] Farsiu [7] Table 1. Results with synthesized sequences. evalution are given in Table 1. The results show, that the block matching with uncertainty analysis (BMU) [9] based registration with Irani-Peleg s [5] fusion algorithm performed best for all test sets. Also, our other registration method, the phase correlation using block processing (BPC), performed well compared to the reference methods. This can be visually confirmed by looking Fig. 1 and Fig. 2 where a zoomed portion of the recontructed images are shown for base images A and B respectively. With the test set D, only BMU method managed to reconstruct the text to a readable form. We also compared the methods with real image sequences captured with a Nokia N90 mobile phone. The phone is based on Series 60 platform with Symbian 8.1a OS. It contains a 220 MHz ARM9 based CPU and a 2M pixel camera with Carl Zeiss optics. We captured five 160 by 120 pixel low resolution images for super resolution image reconstruction. In real sequences there was no ground truth images available. Also, the motion parameters were unknown. The visual inspection was used to evaluate the performance of the methods. Similar results like with synthetic data was achieved. Fig. 3 shows a zoomed portion of the recontructed images for real image test case. The block matching with uncertainty analysis (BMU) [9] registration with Irani-Peleg s [5] fusion algorithm performed very well again. Also, the methods of Zomet [6] and Farsiu [7] produced good results. It should be noted, that only the method of Zomet [6] normalizes the contrast of the image. This process could be done after other methods too. The computational complexity of the algorithms was not compared because our BMU method is implemented with C and only quite slow Matlab software was available for other methods. 4. CONCLUSIONS We have presented two new alternative image registration methods for super-resolution reconstruction. The main idea was to develop efficient solutions which could be used in resource limited platforms such as mobile phones. The proposed methods are based on block matching with uncertainty analysis and phase correlation using block processing. The experiments show that the proposed methods perform well if compared to state of the art approaches and indicate that the methods can be applied in practise. Acknowledgments We thank Peyman Milanfar from UC Santa Cruz for allowing us to use their super-resolution software package. The financial support of the Academy of Finland (project no ) is gratefully acknowledged.

5 (a) (b) (c) (d) (e) (f) Fig. 3. Experiments with real image data: (a) LR image (b) Vandewalle (c) BMU (d) BPC (e) Zomet (f) Farsiu 5. REFERENCES [1] S. C. Park, M. K. Park, and M. G. Kang, Super-resolution image reconstruction: a technical overview, IEEE Signal Processing Magazine, vol. 20, no. 3, pp , May [2] R. Y. Tsai and T. S. Huang, Multiframe image restoration and registration, in Advances in Computer Vision and Image Processing, JAI Press, 1984, vol. 1, pp [3] H. Foroosh, J. B. Zerubia, and M. Berthod, Extension of phase correlation to subpixel registration, IEEE Transactions on Image Processing, vol. 11, no. 3, pp , March [4] P. Vandewalle, S. Süsstrunk, and M. Vetterli, A Frequency Domain Approach to Registration of Aliased Images with Application to Super- Resolution, EURASIP Journal on Applied Signal Processing (special issue on Super-resolution), vol [5] M. Irani and S. Peleg, Improving resolution by image registration, CVGIP: Graphical Models and Image Processing, vol. 53, no. 3, pp , May [10] C. Tomasi and T. Kanade, Detection and tracking of point features, Tech. Rep. CMU-CS , Carnegie-Mellon University, [11] P. Aschwanden and W. Guggenbühl, Experimental results from a comparative study of correlationtype registration algorithms, in Robust Computer Vision: Quality of Vision Algorithms, W. Förstner and St. Ruwiedel, Eds., pp Wichmann, Karlsruhe, [12] J. Hannuksela, P. Sangi, and J. Heikkilä, Visionbased motion estimation for interaction with mobile devices, Vision for Human-Computer Interaction, A special issue of the journal on Computer Vision and Image Understanding, [13] M. A. Fischler and R. C. Bolles, Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography, Communications of the ACM, vol. 24, pp , [14] C. D. Kuglin and D. C. Hines, The phase correlation image alignment, in IEEE Int. Conf. Cybernetics and Society, Sept 1975, pp [6] A. Zomet, A. Rav-Acha, and S. Peleg, Robust superresolution, in International Conference on Computer Vision and Pattern Recognition, Dec 2001, vol. 1, pp [7] S. Farsiu, M. D. Robinson, M. Elad, and P. Milanfar, Fast and robust multiframe super-resolution, IEEE Transactions on Image Processing, vol. 13, no. 10, pp , [8] F. Šroubek and J. Flusser, Resolution enhancement via probabilistic deconvolution of multiple degraded images, Pattern Recognition Letters, vol. 27, no. 4, pp , March [9] P. Sangi, J. Hannuksela, and J. Heikkilä, Global motion estimation using block matching with uncertainty analysis, in Proc. 15th European Signal Processing Conference, Poznan, Poland, 2007, p. 5.