Frame Rate Up Conversion Via Bayesian Motion Estimation

Transcription

1 Frame Rate Up Conversion Via Bayesian Motion Estimation Yue Wang* a, Siwei Ma b, Wen Gao b a Graduate University o Chinese Academy o Sciences, Beijing,100080, China; b Institute o Digital Media, Peking University, Beijing, ,China {wangyue,swma,wgao}@jdl.ac.cn ABSTRACT In this paper, a novel block-based motion compensated rame interpolation (MCI) algorithm is proposed to enhance the temporal resolution o video sequences. We ormulated motion estimation into MAP ramework, and solved it via Bayesian belie propagation. By eectively incorporating a priori knowledge o the motion ield and optimizing the whole motion ield synchronously, it could derive more accurate motion vectors than traditional methods. Finally, adaptive overlapped block motion compensation (OBMC) is used to reduce blocking artiacts. Experimental results show that the proposed method outperorms other methods in both objective and subjective quality. Keywords: rame interpolation, rame rate up conversion, motion estimation 1. INTRODUCTION Frame interpolation, which is also reerred to as rame rate up conversion (FRUC), is a video processing technique used in various applications. The most practical one is video ormat conversion [1]. For example, old television is typically ilmed at 30 or 60 rames/s, but high deinition television (HDTV) has a reresh rate o 60Hz or 120Hz. To play the old videos on the new display devices, we need to convert the lower rame rates to higher ones. Besides, rame interpolation can also be used in low bit rate video coding [2].With low bandwidth limits, one can send the ull rame rate video at the cost o introducing annoying artiacts. Alternatively, we could reduce the rame rate by hal so that each rame has better quality. In the latter case, we need to perorm FRUC at the decoder to display the video at a ull rame rate. Conventional FRUC methods usually used simple rame repetition or rame averaging. However, since these methods didn t take motion inormation into account, the interpolated rame would have annoying artiacts such as motion jerkiness and blurring. To reduce these artiacts, motion compensated rame interpolation (MCFI) is widely adopted in recent FRUC methods. The most important thing to be considered in MCFI is the accuracy o estimated motion vectors, since it determines the quality o interpolated rames. Most MCFI methods utilize the block matching algorithm (BMA) to perorm motion estimation (ME), since it s simple and easy to implement. Although BMA has been used successully in video compression, however, in FRUC the task o ME is to get the true motion ield rather than minimizing the residue energy. So a lot o eorts have been made to improve the block-based motion estimation methods. Most o them are imposing a smoothness constraint on the motion iled, since it s widely accepted that true motion ield should be smooth except along motion boundaries. The most amous and successul method may be 3D-RS block matcher proposed by De Haan[3]. Rather than taking all the possible candidate vectors into account, 3DRS algorithm calculates on a small number o spatial and temporal prediction vectors rom a 3-D neighborhood, which makes it eicient in yielding coherent vector ields. In [4], Zhai proposed to use overlapped motion estimation (OBME) to avoid alling into local optimal solution. Besides, many methods adopt various post-processing to correct the unreliable motion vectors ater BMA. For example, MV-median iltering is widely used in many FRUC schemes, since it is eective in suppressing single outliers and preserving edges. In [5], Huang proposed to detect and correct unreliable motion vectors according to residual energy and MV correlation. However, motion vectors obtained by these methods are still unreliable in many cases. Although 3DRS or OBME takes motion coherence into account, however, the cost unctions are still on a inite support around current block and they may be easily trapped into sub-optimal solutions when multiple motion vectors may lead to small SADs. Furthermore, in Visual Communications and Image Processing 2010, edited by Pascal Frossard, Houqiang Li, Feng Wu, Bernd Girod, Shipeng Li, Guo Wei, Proc. o SPIE Vol. 7744, 77442L 2010 SPIE CCC code: X/10/$18 doi: / Proc. o SPIE Vol L-1

2 the area where complicated motion or deorming happens, there is usually a group o wrong MVs. In this case, neighboring candidates or median iltering will cause error propagation and make the case worse. In this paper, we present a true motion estimation method, which ormulates motion estimation into MAP ramework. We set up a global cost unction or the whole motion ield. A priori knowledge o the motion ield is eectively incorporated into the cost unction. Then the cost unction is minimized by belie propagation. Finally, adaptive OBMC is used to reduce blocking artiacts o the area which contains complicated motions or deorming objects. It dierentiates rom previous methods by optimizing the whole motion ield synchronously, thus local optimum could be avoided. Experimental results show that the proposed method outperorms others in both objective and subjective quality. The rest o the paper is organized as ollows. In Section 2, the proposed algorithm or FRUC is presented. Experimental results and analysis are in section 3. Finally, section 4 concludes this paper. 2.1 Bayesian ormulating 2. PROPOSED METHOD BMA usually takes sum o absolute dierence (SAD) as the matching criterion. For the ( i, j) th block B (, ) t 1/2 i j in the interpolated rame It 1/2, the bidirectional SAD o motion vector V is computed as: SAD( Bt 1/ 2( i, j), V ) It( X s V ) It 1( X s V ) (1) S Bt 1/2 (, i j) X s is the coordinates o pixel S, which is located within block B (, ) t 1/2 i j. It () and I () t 1 represents the intensity o pixel located at ( ) in previous rame and next rame respectively. This is also reerred to as bidirectional motion estimation, which has superiority in handling hole problem and overlapping problem (illustrated in Figure 1). ( vv, ) x y ( x, y) (,) vv x y t t 1/2 t 1 Figure 1. bidirectional motion estimation Proc. o SPIE Vol L-2

3 Under the assumption that motion compensated dierences ollow independent Laplacian distribution, it s proved that the block-based motion estimator which minimizes SAD is equivalent to a ML estimator [6]. arg max PI (, I) arg min SADB ( ( X ), V) (2) ML t 1 t t 1/2 p p p H In the upper equation, H is the set o blocks in the interpolated rame It 1/2. X p is the block-based coordinates (, i j ) o block p, and V p is its estimated motion vector. is the motion ield which consists o all the estimated MVs in It 1/2. This matching criterion is usually unreliable or true motion, and is quite sensitive to noises. As mentioned in Section 1, smoothness prior o the motion ield could be incorporated to help get more accurate motion vectors. In this work we ormulate motion estimation into MAP ramework which maximizes the posterior probability o motion ield : PI (, I ) P(, I ) arg max P( I, I ) arg max arg max( P( I, I ) P(, I )) (3) t t 1 t 1 MAP t t 1 t t 1 t 1 PI (, 1) t It The irst term is the likelihood unction, which is in the same orm as (2). For the second term, a common model to describe the prior probability is the Gibbs distribution: 1 P(, I) exp( U( )) (4) Z U( ) is called an energy unction, which represents a prior constraint on the motion ield. As it s widely accepted that true motion ield should be smooth except along motion boundaries, so here we deine U( ) as: (5) U W V V V V DISC K 2 ( ) ( p, q) min(( p q), _ ) pq, N pq, N N stands or the our-connected neighborhood. Large dierence between motion vectors o neighboring blocks is penalized in order to keep a smooth motion ield. Meanwhile, the truncated actor DISC _ K allows o large discontinuity along motion boundaries. Rewriting (3) to negative logarithm orm, we get: (6) arg min( WV (, V) D( V)) MAP p q p p pq, N p H Where Dp( VP) SAD( Bt 1/ 2( X p), Vp), is a controlling actor, which is set as 1/30 experientially in this method. 2.2 Belie propagation n Cost unction o (6) is a global cost o the whole motion ield, and minimizing it has a complexity o Ok ( ), where k is the number o candidate motion vectors and n is the number o blocks. It s an intolerable complexity or real time application. In this work, we solve it using belie propagation, which has been demonstrated to be a powerul tool to solve MRF problems in recent years [7][8][9]. Belie propagation works by passing local messages around the graph. Each message is a k dimension vector. The message sent rom block p to block q at time t is deined as: m V W V V D V m V (7) t t 1 p q( q) min( ( p, q) P( P) s p( p)) Vp s N( p)\ q N ( p) \ q 0 denotes the neighboring blocks o p except q. Message mp q( Vq) is initialized as 0. Ater T iterations when the messages converged, we choose the inal motion vectors which minimize the lowing cost or each block q : Proc. o SPIE Vol L-3

4 b ( V ) D ( V ) m ( V ) (8) T q q q q p q q p N( q) It s proved that the solution obtained through minimizing (8) corresponds to the optimal solution o (6), but the 2 complexity is reduced to ONTk ( ). BP works by its powerul message passing. Each message represents the probability o the motion that receiver should be going along according to all inormation rom the sender up to the current iteration. There is a magic property o BP: the inluence o messages between largely diverged blocks will all o quickly. So it could easily tell apart objects with dierent motions. With a recursive update process, the messages will pass through and connect the whole motion ield. There is a problem that which blocksize is appropriate. Generally, a large blocksize has less computing and converges ast, but i a single block contains several object motions, the interpolated one will be blocky and dirty. On the other hand, a small block size may reduce block artiacts, but it will cause a slow convergence. Considering these acts and based on lots o experiments, we suggest blocksizes o 4 4or small rame size like qci, 8 8or ci, and or 4ci and larger sizes. 2.3 Adaptive OBMC Ater all the motion vectors are determined ater BP, interpolation is then perormed by averaging two bidirectional motion compensated block. For each pixel s in block p, its interpolated value could be represented as: where w and I ( X ) w I ( X V ) w I ( X V ) (9) t 1/ 2 s t s p b t 1 s p w b are the weights o orward and backward predictions and are both simply set as 1/2. However, the block based motion estimation only works well under linear translational motion model. I the block contains complicated motion, such as rotating and deorming, or it has objects with dierent motions, the estimated motion vector would be unreliable or the pixels in the block and would cause annoying block artiacts. In this case, we irstly check whether the obtained motion vector is reliable based on its bidirectional SAD. I SAD( Bt 1/2( X p), Vp) blocksize 5 (10) then V p is considered as unreliable or block p, and OBMC is applied to this block. For the sake o simpler notations, we assume that pixel s is located at upper let quadrant o block p, then it will be interpolated using motion vectors o the block itsel, the upper block, the let block and the upper let block: It 1/2 ( Xs) wm, n( Xs) ( It( Xs V( i m, j n)) It 1( Xs V( i m, j n)) (11) 2 m 0 n 0 wmn, ( Xs) is weighting coeicients determined by the relative position o pixel s within block p, and ( i, j ) is the coordinates o block p. In this work, bilinear window[10] is used or the weighting window w, ( X ). The interpolation rules or pixels in the other quadrants can be derived in a similar way. Figure2 shows one o the interpolated rames in oreman.ci beore and ater OBMC. It could be seen that block artiacts are successully reduced. mn s 3. EXPERIMENTAL RESULTS In order to veriy the validity o the proposed method, we test it using various sequences. We also compared our method with two other FRUC methods. The irst is 3DRS ME, and the second is the FRUC scheme with OBME, median iltering and OBMC. In all the experiments, the block size is set to 8 8 and search range is set to 16. Iterative message updating in BP is stopped ater 5 iterations. Proc. o SPIE Vol L-4

5 (a) (b) Figure 2. interpolated rame o oreman.ci (a)beore OBMC (b) ater OBMC 3.1 Objective Evaluation Firstly, we compare the PSNR perormance o the proposed method with the other two methods. We skipped the odd rames and interpolated them by the even rames using dierent ME methods. Then PSNR is computed between the original rame and the generated rame. The results are given in Table 1. All o the test sequences are in ci ormat and interpolated rom 15 rames/s to 30rames/s. It s could been seen that the proposed method provides better PSNR perormance than other two methods. Table 1 comparison o PSNR perormance Sequence Proposed method 3DRS ME OBME, median iltering and OBMC oreman ootball mobile bus Paris carphone lower tempete Subjective Evaluation Sometimes the objective evaluation such as PSNR measures could not accurately represent the real quality o the interpolation. Figure 3 shows the visual comparison o the interpolated pictures o oreman, ootball, mobile and entrapment_tokyo_towers. Proc. o SPIE Vol L-5

6 (a) (b) (c) (a) (b) (c) (a) (b) (c) (a) (b) (c) Figure 3. subjective comparison o the methods: (a)3drs ME. (b)obme, median iltering and OBMC (c) proposed method Proc. o SPIE Vol L-6

7 From the comparison we could see that our method achieves obvious better visual quality than the other two methods. In the area where complicated motion happens (the mouth area o oreman and the players o ootball), our method has better perormance. For the blocks where multiple motion vectors may lead to small SADs (the calendar o mobile and the tower o last sequence), other methods have a dirty interpolation. The median iltering even makes the case worse, since the motion vectors o neighboring blocks are unreliable as well. However, the proposed method is robust to these cases. The message passing process could connect the whole motion ield and converge to the global optimal solution. 3.3 Convergence To speed up the iterative convergence, we utilized accelerated BP [11], in which the message is used immediately ater update in each iteration. Table 2 shows the PSNR o oreman in each iteration and it s calculated beore OBMC. Generally, there would be little PSNR improvement ater 3-4 iterations. Table 2 PSNR o oreman in each iteration nth iteration PSNR CONCLUSION In this paper, a novel block-based ME method is proposed or FRUC. We ormulate motion estimation into MAP ramework. A global cost unction o the whole motion ield is set up and then minimized by belie propagation. Adaptive OBMC is also utilized to reduce block artiacts. This method is able to derive true motions, especially in regions which contain periodic texture or complicated motions. Experimental results demonstrate that our method outperorms others in both objective and subjective quality. 5. ACKNOWLEDGMENT This work was supported in part by National Science Foundation ( ) and National Basic Research Program o China (973 Program, 2009CB320903). REFERENCES 1. O.A.Ojo, G de Haan, Robust motion-compensated up conversion, IEEE Trans. Consum. Electron., vol. 43, no. 4, pp , Nov, G. Dane and Truong Nguyen., Quality enhancement in standard compliant rame rate up conversion by motion smoothing, in Proc. EUSIPCO, G. de Haan, P. W. Biezen, H. Huijgen, and O. A. Ojo, True-motion estimation with 3-D recursive search block matching, IEEE Trans. Circuits Syst. Video Technol., vol. 3, no. 5, pp , Oct J. Zhai, K. Yu, J. Li, and S. Li, A low complexity motion compensated rame interpolation method, in Proc. IEEE ISCAS, May 2005, pp Ai-mei Hhuang and Truong Nguyen Correlation-Based Motion Vector Processing With Adaptive Interpolation Scheme or, IEEE Transactions on Image Processing, vol. 18, issue 4, pp Ioannis Patras, Emile A. Hendriks, Reginald L. Lagendijk: Probabilistic Conidence Measures or Block Matching Motion Estimation. IEEE Trans. Circuits and Systems or Video Technology 17(8): (2007) 7. W.T.Freeman, E. Pasztor, and O. Carmichael. Learning low level vision. Int. J. o Computer Vision, 40:25 47, J. Sun, N.N. Zheng, and H.Y. Shum. Stereo matching using belie propagation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(7): , Proc. o SPIE Vol L-7

8 9. Pedro F. Felzenszwalb and Daniel P. Huttenlocher. Eicient Belie Propagation or Early Vision. International Journal o Computer Vision, Vol. 70, No. 1, October M.T. Orchard and C. J. Sullivan, "Overlapped block motion compensation: An estimation-theoretic approach," IEEE Trans. Image Processing, vol. 3, no. 9, pp , Sept Tappen,M.F. Freeman,W.T. Comparison o graph cuts with belie propagation or stereo, using identical MRF parameters ICCV2003, vol Oct Proc. o SPIE Vol L-8