MULTI-CHANNEL COMBINED LOCAL GLOBAL OPTICAL FLOW APPROACH USING PHOTOMETRIC INVARIANTS

Transcription

1 MULTI-CHANNEL COMBINED LOCAL GLOBAL OPTICAL FLOW APPROACH USING PHOTOMETRIC INVARIANTS Sarah Elsharkawy, Haythem Elmessiry, and Mohamed Roushdy Computer Science Department Faculty of Computer and Information Sciences, AinShams University, Cairo, Egypt Abstract A fundamental problem in the processing of image sequences is the measurement of optical flow (or image velocity). The goal is to compute an approximation to the 2-D motion field which is the projection of the 3-D velocities of surface points onto the imaging surface from spatiotemporal patterns of image intensity. Optical flow techniques can be classified into local approaches which are robust under noise, global approaches that yield dense flow fields, and finally into combined local global approaches that combine the advantages of both the local and the global approaches. Unfortunately, most the previous approaches assume that a point occurring in the scene has the same intensity in every frame. However, in real-life applications this is not the case where different changes and shadows may occur. Therefore, Photometric invariants were proposed by Y.Mileva to solve this problem by applying them to a multi-channel global approach which resulted in a better robustness under changes. Hence, we proposed to extend this work by applying photometric invariants to a multi-channel combined local global approach aiming to formulate a model that is robust under noise and changes and also provides dense flow fields. Experimental Results showed that the proposed method significantly performs better than the multi-channel global approach. Keywords: Optical Flow, Computer Vision, Motion Estimation, Illumination variations, Photometric invariants. 1 Introduction The estimation of motion in images is a basic task in computer vision with many interesting applications. A primary goal of the field is to estimate the scene or object motion as precisely as possible. Most of the optical flow techniques are based on the brightness constancy assumption (BCA) which states that an image pixel, representing a point on an object, does not change its brightness (grey-level) value from an instant of

2 time to the next instant of time. Let be the continuous space-time intensity function, where x and y are the x- and y-coordinates of the image point, respectively, and t denotes the time. Now let us suppose that the point moves by δx, δy in time δt to. According to the above mentioned assumption, we assume that the intensity remains constant along a motion trajectory and therefore This assumption usually holds true if δx, δy, and δt are not too large. However, in a realistic scene this is never the case. A pixel can change its brightness value because an object moves (translates or rotates) to another part of the scene with different lighting or because the of the scene (globally or locally) changes in time. And therefore, all techniques based on the BCA deteriorate under image sequences having variations which violate the assumption. In order to formulate a model that can be able to effectively estimate the optical flow velocities under variations, we need to replace the BCA with other assumption that can cope with the brightness changes. Y.Mileva [3] proposed to replace the BCA with one of the photometric invariants constancy assumptions. Photometric invariants with respect to shadows, shading and lighting can be derived easily from the RGB images. These Photometric invariants are derived from the HSI color space [6, 12], from normalized RGB channels [6], or from the representation that is obtained via the spherical coordinate transform (SCT) [12]. These expressions are in general invariant under changes of multiplicative and/or additive type. All optical flow techniques that depend on the BCA in their computation use only one channel which is the grey-level channel, while in case of using photometric invariants we need one or more channels according to the type of invariant used. For example, the log-derivatives technique requires 6 channels, the normalization technique requires 3 channels, and so on. Therefore, there is a need to design a multi-channel approach that can be able to handle more than one channel in the optical flow estimation. Yana Mileva [3] introduced a modification to the original Horn and Schunck [7] optical flow by changing it into a multi-channel approach and then used it to test the performance of the photometric invariants. The original Horn and Schunck approach [7] which is a global technique has the advantage of providing dense flow fields (i.e.: estimating optical flow velocities for every single pixel) but it has the disadvantage of being sensitive to noise. A. Bruhn and J.Weickert [2] introduced a combined local global (CLG) technique

3 that is able to estimate dense flow fields and is also robust under noise. Therefore, we propose to modify the CLG approach into a multi-channel CLG approach and test the performance of photometric invariants using it. 2 Methodology Before deriving a formulation for our optical flow method, we give an intuitive idea of the techniques included in such a model. Let us consider an image sequence, where denotes the location within a rectangular image domain Ω and t 0 denotes time. The resulting flow field is denoted by between two consecutive frames and. Horn and Schunck approach minimizes the energy functional Where serves as a regularization parameter. Larger values for α result in a stronger penalization of large flow gradients and lead to smoother flow fields. The combined local global approach is similar to the Horn and Schunck approach except that it uses a Gaussian Kernel with a standard deviation that was introduced by Lucas and Kanade [11] in their local technique. The CLG approach minimizes the following energy functional Although the idea of CLG is quite simple it constitutes an important progress, as both the Lucas-Kanade method and Horn-Schunck method can be seen as special cases of CLG with certain parameter settings. For the Lucas-Kanade technique the parameter α is zero, while for the Horn-Schunck approach the parameter ρ is zero. In order to formulate the multi-channel versions of the previous techniques, let us assume that we have different channels numbered by denoted by. The multi-channel Horn and Schunck approach proposed by Y.Mileva minimizes Where serves as a weight that differentiates the importance of the different channels. Similarly, we propose the multi-channel CLG approach to minimize the following energy functional

4 It is worth knowing that if N=1 and γ=1 then it will act exactly as the single channel CLG approach, and if ρ=0 then it will act exactly like the multi-channel Horn and Schunck approach. 3 Implementation In order to render our approach more robust against outliers in both the data and smoothness term, we propose the minimization of the following functional Where is a convex function that ensures the existence of a solution. Due to the small positive constant, is still convex which offers advantages in the minimization process. Moreover, this choice of does not introduce any additional parameters, since is only for numerical reasons and can be set to a fixed value, which we choose to be In order to minimize the nonlinear functional (6), we must figure out their corresponding Euler-Lagrange equations. Using the calculus of variations we obtain the following nonlinear partial differential equations: Where and is an abbreviation for The preceding Euler-Lagrange equations (7, 8) are non-linear in their arguments. In order to reach a linear system of equations, which can be solved with common numerical methods, we have to use two nested fixed point iterations and a coarse-to-fine warping strategy. The rest of the minimization and numerical approximations are done exactly as in [4].

5 4 Experimental Results In the first experiment, we compare the results of our proposed multi-channel CLG approach with the multichannel Horn and Schunck approach under varying using the different photometric invariants. Let us start our evaluation with the RubberWhale sequence available at flow/data/. The images of the sequence are of size 584x388. We used frame 10 and 11 in our experiments. Frame 10 and 11 are shown in Figure 1. (a) (b) Figure 1: (a) Frame 10 of the RubberWhale Sequence. (b) Frame 11 of the RubberWhale sequence Also we created a second version of frame 11 using a Gaussian additive (Figure 2) in order to be able to test our proposed approach under varying. The ground truth motion between frame 10 and 11 is available at bury.edu/ flow/data/ as seen in Figure 3. The estimated motion between frame 10 and 11 using the multi-channel CLG approach with the Log Derivatives constancy assumption is shown in figure 4. Figure 5 shows the estimated motion between frame 10 and the degraded version of frame 11 using the same approach. Figure 2: Frame 11 of the RubberWhale sequence with additive Gaussian The different photometric invariants and the corresponding results in terms of the average angular error (AAE) are listed in table 1. Where the Angular Error is the angle between the correct and the estimated flow vectors and it is calculated as follows

6 , where denotes the correct flow and is the estimated flow. (a) (b) Figure 3: (a) Ground truth motion between frame 10 and 11. (b) Direction-to-color map. The same experiment was done on the Hydrangea, Yosemite and Dimetrodon image sequences and their results are listed in tables 2, 3, and 4 respectively. The Hydrangea images are of size 584x388, The Yosemite images are of size 316x252, and the Dimetrodon images are of size 584x388. In the illustrated results, all the weights γ i have been set to one and the standard deviation of the Gaussian kernel ρ is set to one. Figure 4: The estimated motion between frames 10 and 11. Figure 5: The estimated motion between frame 10 and frame 11 with Gaussian.

7 Table 1. Results of the RubberWhale image sequence Average Angular Errors Multi-channel Horn and Multi-channel CLG Concept # channels Schunck Approach Approach Gray Level RGB Color Space HSV Color Space 1 (Hue) RΦΘ Color Space 2 (Φ, Θ) Arithmetic ized Mean Log Derivatives Table 2. Results of the Hydrangea image sequence Average Angular Errors Multi-channel Horn and Multi-channel CLG Concept # channels Schunck Approach Approach Gray Level RGB Color Space HSV Color Space 1 (Hue) RΦΘ Color Space 2 (Φ, Θ) Arithmetic ized Mean Log Derivatives As can be seen from the results, the proposed Multi-channel CLG approach performs better than the Multichannel Horn and Schunck approach in almost all the cases. The results also ensure that both the grey-level constancy assumption and the RGB constancy assumption fail under varying. On the other hand, the photometric invariants constancy assumptions are almost not affected by the changes.

8 Table 3. Results of the Yosemite image sequence Average Angular Errors Multi-channel Horn and Multi-channel CLG Concept # channels Schunck Approach Approach Gray Level RGB Color Space HSV Color Space 1 (Hue) RΦΘ Color Space 2 (Φ, Θ) Arithmetic ized Mean Log Derivatives Table 4: Results of the Dimetrodon image sequence. Average Angular Errors Multi-channel Horn and Multi-channel CLG Concept # channels Schunck Approach Approach Gray Level RGB Color Space HSV Color Space 1 (Hue) RΦΘ Color Space 2 (Φ, Θ) Arithmetic ized Mean Log Derivatives In the second experiment, we investigate the robustness of our approach under noise in comparison with the multi-channel Horn and Schunck approach. We use the RubberWhale image sequence, but here we created more than one degraded version of frame 11 using Gaussian noise of different standard deviations. Figures 7 and 8 illustrate the resulting estimated motion under noise. The results in terms of the average

9 angular error are listed in table 5 which shows that the proposed Multi-channel CLG approach appears to be more robust under noisy structures than the Multi-channel approach. Figure 6: Frame 11 of the RubberWhale sequence severely degraded by Gaussian noise with σ n =40 Table 5. Average angular errors of the second Experiment. Average Angular Errors σ n Multi-channel Horn and Schunck Multi-channel CLG approach approach Figure 7: Estimated motion under Gaussian noise with standard deviation σ n =0. Figure 8: Estimated motion under Gaussian noise with standard deviation σ n =40.

10 5 Discussion and Conclusion We have presented in this paper a new approach for estimating optical flow when image sequences are recorded under conditions of varying. To achieve this, we combined the multi-channel Horn and Schunck approach introduced by Y.Mileva and the CLG approach together into a multi-channel CLG approach. Then we replaced the gray-level constancy assumption with the photometric invariants constancy assumption. As a result, our method is robust under changes and noise and also provides dense flow fields. The experimental results reported in this paper show the superiority of our method when changes significantly from one image to the next. We have shown this via statistical comparisons on a set of image sequences for which the ground truth is known. Furthermore, we have presented results that prove the robustness of our proposed multi-channel CLG approach under noise in comparison with the multichannel Horn and Schunck approach. 6 Future Work In the Future, we would like to investigate the effect of using the gradient constancy assumption together with the Brightness constancy assumption under changes, and the effect of applying a Gaussian smoothing kernel on the gradient constancy assumption as well. Furthermore, we are going to formulate a multi-channel version of it and test its performance using the photometric invariants. References [1] S.A. Shafer. Using color to separate reflection components. COLOR research and application, 10(4): , Winter [2] Bruhn, J. Weickert, and C. Schnörr. Lucas/Kanade meets Horn/Schunck: Combining local and global optic flow methods. Int. J. Comput. Vis., 61(3): , [3] Y. Mileva, A. Bruhn, and J. Weickert. Illuminaion-Robust variational optical flow with Photometric Invariants. In Lecture Notes in Computer Science, volume 4713, pages Springer [4] T. Brox, A. Bruhn, N. Papenberg, and J.Weickert. High accuracy optic flow estimation based on a theory for warping. In T. Pajdla and J. Matas, editors, Computer Vision ECCV 2004, volume 3024 of Lecture Notes in Computer Science, pages Springer, Berlin,2004. [5] Bruhn, J.Weickert, T. Kohlberger, and C. Schn orr. A multigrid platform for real-time motion computation with discontinuity-preserving variational methods. International Journal of Computer Vision, 70(3): , Dec.2006.

11 [6] P. Golland and A. M. Bruckstein. Motion from color. Computer Vision and Image Understanding, 68(3): , Dec [7] B. Horn and B. Schunck. Determining optical flow. Artificial Intelligence, 17: ,1981. [8] R.J. Andrews and B. C. Lovell. Color optical flow. In Eds. Proceedings Workshop on Digital Image Computing, Brisbane, [9] J. Lim, J. Ho, M.-H. Yang, and D. Kriegman. Passive photometric stereo from motion. In Proc. IEEE Int. Conf. Computer Vision, [10] Lucas, B.D Generalized image matching by the method of differences. PhD thesis, School of Computer Science, Carnegie Mellon University, Pittsburgh,PA. [11] Lucas, B. and Kanade, T An iterative image registration technique with an application to stereo vision. In Proc. Seventh International Joint Conference on Artificial Intelligence, Vancouver, Canada, pp [12] J.van Weijer and T. Gevers. Robust optical flow from photometric invariants. In Proc IEEE International Conference on Image Processing, volume 3, pages , Singapore, Oct.2004.