Foreground-Object-Protected Depth Map Smoothing for DIBR

Transcription

1 01 IEEE Internatonal Conference on Multmeda and Expo Foreground-Object-Protected Depth Map Smoothng for DIBR Xao-han Lu, Fang We, Fang-mn Chen School of Informaton and Communcaton Engneerng Bejng Unversty of Posts and Telecommuncatons Bejng, Chna Abstract Depth mage based renderng (DIBR) uses a -D color mage and ts assocated depth map to render vrtual vews and then a stereoscopc mage. One of the man problems n DIBR s how to reduce the sze of holes n the generated vrtual vew mage. Smoothng the depth map before mage warpng s a common soluton. However, the prevous smoothng methods brng edge dstortons to the foreground objects whch are more valuable than the background. In ths paper, a pecewse smoothng flter s adopted, performng flterng on the depth map only n the background regons whle leavng the foreground regons unchanged. And the flterng s also controlled by the regstraton ponts of the depth map. Expermental results show that the proposed method obtans better subjectve qualtes for the vrtual vews, and t s more tme-savng compared to the former methods whle havng a better PSNR performance. Keywords-DIBR; depth map smoothng; foreground object protecton; depth map regstraton I. INTRODUCTION Stereoscopc vdeos are gettng more and more popular wth the rapd developments of dgtal technologes. In most stereoscopc vdeo systems, two temporally synchronzed vdeo streams (respectvely for the left and rght eye) are used to provde the stereoscopc percepton to the vewers. Such a stereoscopc vdeo needs much more storage spaces and bandwdth than a monocular one. Besdes, t can only provde unalterable stereoscopc perceptons, whch are not enough for some applcatons, such as, a stereoscopc nteractve applcaton provdng users wth the freedom to dynamcally customze the stereoscopc perceptons to meet ther own appettes. Vdeo-plus-depth stereoscopc systems wth lower storage spaces, bandwdth requrement and customzable stereoscopc perceptons turn out to be a new choce, n whch the depth mage based renderng (DIBR) s one of the key algorthms, to synthesze vrtual vews from a monocular mage and ts assocated depth mage. The qualty of the fnal stereoscopc vdeos largely depends on the effectveness of DIBR results. DIBR algorthms are bascally dvded nto three steps, depth mage smoothng, mage warpng, and holes fllng. The holes, represent the areas n the vrtual vews whch are not vsble n the orgnal monocular mage. Smoothng the depth map to reduce ts dscontnuty s helpful to decrease the sze of the holes, thus makng holes much easer to be flled. In the prevous works, Gaussan flter s usually used as the smoothng flter. For example, [] utlzed a symmetrc Gaussan low-pass flter to smooth the depth mages. [3] and [9] used an asymmetrc Gaussan low-pass flter. Edge dependent low-pass flters were proposed n [5] and [10]. [7] adopted parallax maps to render orgnal mages. These smoothng flters successfully preclude dstortons such as rubber-sheet artfacts and strp-lke mparments. However, all of them blur the edges n depth maps, resultng n that both foreground and background pxels are used to fll the holes n the generated vrtual vew mages. In ths paper, a new approach to smooth the depth maps for DIBR s presented. It s based on the concept of foreground object protecton (n secton ). After the sharp depth dscontnutes predcton, the depth map s smoothed wth the prncple that not changng pxel values of any foreground regons, but only that of the background. In ths way, the proposed method produces stereoscopc mages wth better subjectve qualty and meanwhle turns out to provde a good tradeoff between the tme savng, the hole reducton, and the PSNR. II. CONCEPT OF FOREGROUND OBJECT PROTECTION A. Foreground Object Defnton Fg. 1(c)(d)(e) show the DIBR process wthout any depth map smoothng. The sharp depth dscontnuty (the boundary n Fg. 1(c)) wthn the depth map results n wder holes n the vrtual vew mage, as n Fg. 1(d). To model the dscontnutes n depth maps, we defne two relatve concepts, foreground and background pxels. A pxel wth hgher depth value s defned as a foreground pxel and a background pxel s defned n the other way round. Accordngly, a foreground and background object can be defned n the same manner. For example, n Fg. 1, the man's head s consdered to be a foreground object, and the wall behnd hm s a background object. B. Problems of Prevous Works n Foreground Object Protecton Prevous depth map smoothng works are not good enough n foreground object protecton. They dstorted the foreground objects more or less. For nstance, Fg. 1(f)(g)(h) /1 $ IEEE DOI /ICME

2 show a DIBR process wth a horzontal Gaussan flter. The smoothng step blurs the edges of the depth map (edge gradents n Fg. 1(f)), and makes the holes n the generated vrtual vew mage thnner and wdely spread (Fg. 1(g)). Ths s because that the foreground pxels close to the holes n the vrtual vew generated wthout smoothng flters are shfted to fll parts of the holes. After the hole-fllng, these foreground pxels are usually coped to completely fll the holes. Then the foreground objects wll be prolonged (Fg. 1(h)). color mage than those n the depth map. Moreover, edges n the color mage are more closely modeled as havng a "ramp lke" profle. As can be seen from Fg. (d), colors of the paw's edge gradually change as a ramp. On the contrary, the depth map usually has a steep edge (see Fg.1(c)) whose correspondng poston n the color mage s usually at the mddle of the ramp. Thngs above make the foreground objects separated by holes after warpng, Fg. (e). Fllng these holes wll make the foreground objects prolonged and dstorted. Hence, to protect the foreground object, the whole edge ramp should be classfed as foreground, whch means, edges of the depth map should be regstered to the poston where color mage ends the edge ramps. (c) (c) (d) (e) (f) (g) (h) Fgure 1. Orgnal color mage Part of the mage, man s head (c) Orgnal depth map (d) Generated vrtual vew mage(holes are marked black). (e) Hole-fllng result for (d). (f) Depth map preprocessed by Gaussan flter (g) Generated vrtual vew mage usng smoothed depth map (h) Hole-fllng result for (g) C. Sgnfcance n Foreground Object Protecton Choosng approprate pxels to fll holes s the key to determne the stereoscopc vsual qualty [8]. Subjectvely, we would prolong the background wndow rather than the man's head to fll the holes. That looks more natural. What's more, the consstence of the foreground objects between the orgnal mage and the generated vrtual vew mage ensures the qualty of stereo percepton. Objectvely, vewers are more senstve to mage qualty degradaton n the foreground than that n the background [6]. In most cases, foreground objects have more crucal detals whch should not be destroyed. In a word, t s better to use pxels from the background as the flled-n pxels for holes, and not change any pxels n the foreground, that s, protect the foreground objects. D. Problem Formulaton of Depth Map Regstraton In practcal cases, edges of the depth map are not totally matched wth those of the color mage [8]. Fg. (c)(d) compare edges of the depth map and those from the color mage on teddy. The edges are to 4 pxels wder n the (d) Fgure. Image teddy Enlarged depth map of teddy (c)enlarged mage teddy whose edges do not match wth those n the depth map (d)enlarged teddy s paw wth an edge ramp(e) Generated vrtual vew mage III. PROPOSED METHOD Our method manly ams to reduce the foreground object dstortons dscussed n the former secton. Frstly, predct the occurrence of holes by fndng the sharp dscontnutes n the depth map. Then a regstraton of depth map s used to deal wth msalgnment between the color mage and the correspondng depth map. At last, a pecewse smoothng flter s performed only n the background. As a result, the foreground pxels n the depth map are preserved; only the depth pxels n background whch would lead to holes are smoothed; other background pxels are also preserved. A. Hole Regon Predcton The smoothng for the depth map s only done on the depth pxels whch would lead to holes. So t s necessary to fnd such depth pxels before applyng smoothng flters to the depth map. Here, suppose that usng a left-eye vew and ts assocated depth map to generate the rght-eye vew, so that holes wll only appear at the rght sde of the foreground objects. In ths case, a hgh-to-low depth value dscontnuty n the horzontal drecton s what we are lookng for. (e) 340

3 If d(x, satsfes the followng rule, (x, and some of the followng rght pxels wll lead to holes after mage warpng. For that, we mark L d (x, = 1: 1 ( x, = 0 d( x, d( x + 1, θ L d (1) otherwse where s a threshold dependng on the magntude of the depth changes. B. Depth Map Regstraton To better protect foreground objects from the mpact of the msalgned depth map, we regster the foreground regons n the depth map to the poston of the edge endngs n the color mage. To fnd the end of edges n color mage, 1-D horzontal Laplacan operator s employed, as n () and (3). The Laplacan operator computes the second-order dervatve, producng double response to a ramp, and well reflectng the begnnng and the endng of a ramp. f f ( x, = () x f ( x, = f ( x, f ( x + 1, f ( x 1, (3) f(x, s the lumnance value of the color mage at the poston (x,. From each poston found from hole regon predcton on (suppose the number of hole regon predcton poston s M), that s, L ( x, y ) = 1, Z, 1 M d whch usually ndcates the mddle of the edge ramp, go rght n the same row, and search for the maxmum response of the 1-D Laplacan operator wthn a certan range N. It s most lkely to be the endng of an edge ramp; t s also the regstraton pont of the depth edge we are lookng for. r = arg max f ( x + j, y ) j Z,1 j N C. Pecewse Smoothng Method After the depth map s regstered, the scopes of the foreground regons are found, that s, those of the background regons are also determned. Now we can begn the pecewse smoothng along the edge ramps of the foreground to the background. To ensure the foreground objects beng preserved after mage warpng, we set the depth value at each poston found from the hole regon predcton to the fnal regsterng pont to the same value, as below. d ˆ ( x + j, y ) = d( x, y ), j Z, 1 j r (6) So that the poston of pxels from (x, y ) to (x +r, y ) after mage warpng s contnuous as they were n the orgnal mage, because mage warpng s just translaton of orgnal mage pxels wth dsplacement drectly or nversely proportonal to the depth values. (4) (5) Afterwards, we adopt a descent processng to smooth the background regons of the depth map, makng the wdth of holes mnmzed wth approprate depth gradents [8]. Here, a fxed gradent G s used to make the wdth of each hole as even as possble. 1 G = + S S represents the number of steps to fnsh descent. Then the new depth value for the background s updated usng fxed gradent G, and wth S steps. dˆ( x + r + k, y ) = d( x + r, y) k G, (8) k Z,1 k S [ d( x, d( x 1, ] Fgure 3. Example for Depth map smoothng Depth map before smoothng Depth map after smoothng IV. THE EXPERIMENTAL RESULTS To evaluate the proposed approach, we adopt mages 'Intervew' wth pxels, mages 'Teddy' and 'Cones' wth pxels as the test mages. A. Parameter Selecton The threshold of depth dscontnuty detecton ( ) should be large enough to reduce the effects of the gradual changes but also small enough to predct huge dscontnuty n depth maps. A value between 5 and 30 s sutable n our cases. Depth maps wth a bgger sze need a larger threshold. We select =10 n our experments. And the depth dscontnutes are generally not greater than 50, so the value of the smoothng step (S) s set to 5. From the statstcs, the wdth of the edge ramps n color mages wth such a resoluton are not larger than 5, so we set N=5. B. Subjectve Vsual Qualty Fg. 4 shows the comparson of depth maps before and after our proposed smoothng method. Fg. 5 shows the generated vrtual rght-vew mages wth and wthout the depth map regstraton on the mage Teddy. To get a clearer vew, these mages are not appled wth any smoothng methods. We can see that after the regstraton, most of the msalgned foreground pxels are rectfed. (7) 341

4 Fg. 6 shows a comparson of the generated rght vew mages usng our proposed method and the orgnal left vew mage, on the mage 'Teddy'. We can see that the proposed method remans the shape and edge qualty of teddy bear (the foreground object relatve to the wall) the same as t was before. Fg. 7 shows enlarged detaled results on the other two mages. In these mages, our proposed method s compared wth the asymmetrc Gaussan method and the edgedependent method. It can also be seen that our method vsually protects the foreground objects whle asymmetrc Gaussan method dstorts the foreground objects (prolonged face and the cone wrapped to a dfferent angle) and edgedependent method makes the foreground pxels blended wth many dfferent background pxels. Fgure 4. Depth map Teddy before smoothng after smoothng From Table 1, we can see that, the proposed method s more tme-savng compared to the former method whle obtanng a better PSNR performance and resultng n relatvely smaller hole sze. V. CONCLUSION Ths paper presents a foreground-object-protected smoothng method for depth maps n DIBR. The proposed method vsually protects the foreground objects n generated rght vew mages. Moreover, t has a better PSNR performance and relatvely smaller hole sze compared to the prevous works. Better stereoscopc qualty and real-tme DIBR could be acheved by usng ths method. Intervew cone teddy TABLE I. COMPARISON OF PSNR, COMPUTATION TIME AND HOLES Methods PSNR Tme(ms) Hole sze Non-smoothng / Edge-dependent Gaussan Asymmetrc Gaussan proposed Non-smoothng Edge-dependent Asymmetrc Gaussan proposed Non-smoothng Edge-dependent Asymmetrc Gaussan proposed ACKNOWLEDGMENT Ths work was supported by the Fundamental Research Funds for the Central Unverstes of Chna (01RC0136). Fgure 5. regstraton result of mage 'teddy' generated vrtual vew mage generated vrtual vew mage wth depth map regstraton C. Objectve Evaluaton Ths secton compares the PSNR, the computaton tme and the hole sze between the proposed method and the prevous works. Experments are done wth Opencv.0 and Mcrosoft Vsual Studo 008. PSNR can reveal the fdelty of depth cues n the generated rght vews mages. To obtan the PSNR, the generated rght vew mages of cone and teddy use the orgnal rght vew mages as the ground truth whle wthout orgnal rght vew mage, that of ntervew s compared wth the generated rght vew mage of the non-smoothng method. The computaton tme shows the speed mprovement of our depth map smoothng algorthm, and the hole sze shows hole reducton performance helpful for reducng vsual artfacts. REFERENCES [1] C. Fehn, Depth-mage-based-renderng (DIBR), compresson and transmsson for a new approach on 3D-TV, Proc. Internatonal Socety for Optcal Engneerng Conf. Stereoscopc Dsplays and Vrtual Realty Systems XI, San Jose, CA, Vol. 591, pp , Jan [] W. J. Tam, G. Alan, L. Zhang, T. Martn, and R. Renaud, Smoothng depth maps for mproved stereoscopc mage qualty, Proc. SPIE Conf. Three-Dmensonal TV, Vdeo, and Dsplay III, Phladelpha, U.S.A., Vol. 5599, pp , Oct [3] Lang-Hao Wang, Xao-Jun Huang, Mng X, Dong-Xao L, Member, IEEE, and Mng Zhang, An Asymmetrc Edge Adaptve Flter for Depth Generaton and Hole Fllng n 3DTV, IEEE Trans on Broadcastng, Vol. 56, No. 3, pp , Sep 010. [4] Pe-Jun, Lee, Non-geometrc Dstorton Smoothng Approach for Depth Map Preprocessng, IEEE Trans. on Multmeda, Vol. 13, No., pp , Apr 011. [5] E. Ekmekcoglu, M. Mrak, S.T. Worrall and A.M. Kondoz, Edge adaptve up-samplng of depth map vdeos for enhanced freevewpont vdeo qualty, Electroncs letters, Vol. 45, No. 7, pp , Mar 009. [6] Mchael R. Frater, John F. Arnold, and Abedn Vahedan Impact of Audo on Subjectve Assessment of Vdeo Qualty n Vdeoconferencng Applcatons, IEEE Trans. on Crcuts and Systems for Vdeo Tech., Vol. 11, No. 9, pp , Sep 001. [7] Tng-Chng Ln, Hsen-Chao Huang, and Yueh-Mn Huang, Preservng Depth Resoluton of Syntheszed Images Usng Parallax- Map-Based DIBR for 3D-TV, IEEE Trans on Consumer Electroncs, 34

5 Vol. 56, No., pp , May 010. [8] Mchael Feseler, Xaoy Jang, Regstraton of Depth and Vdeo Data n Depth Image Based Renderng, 3DTV-Conference: The True Vson-Capture, Transmsson and Dsplay of 3D Vdeo, Potsdam, Germany, pp. 1-4, May 009 [9] L. Zhang and W. J. Tam, Stereoscopc mage generaton based on depth mages for 3DTV, IEEE Trans. Broadcast., vol. 51, pp , Jun [10] W. Y. Chen, Y. L. Chang, S. F. Ln, L. F. Dng, and L. G. Chen, Ef cent depth mage based renderng wth edge dependent depth lter and nterpolaton, n Proc. IEEE Int. Conf. Multmeda and Expo, Jul.005, pp Fgure 6. Orgnal left vew for mage 'teddy' generated rght vew mage wth the proposed method (c) (d) (e) (f) (g) (h) Fgure 7. Enlarged segments for mage 'ntervew' and 'cone'. (e)orgnal left vew mage, and generated rght vew mage (f) wth the asymmetrc Gaussan smoothng method (c)(g) wth the edge-dependent method (d)(h)wth the proposed method 343