On the Bandwidth of the Plenoptic Function

Transcription

1 IEEE TRANSACTIONS ON IMAGE PROCESSING On he Bandwidh of he Plenopic Funcion Minh N. Do, Day Marchand-Maille, and Marin Veerli Absrac The plenopic funcion (POF) proides a powerful concepual ool for describing a number of problems in image/ideo processing, ision, and graphics. For example, image based-rendering can be seen as sampling and inerpolaion of he POF. In such applicaions, i is imporan o characerize he bandwidh of he POF. We sudy a simple bu represenaie model of he scene where bandlimied signals (e.g. exure images) are pained on smooh surfaces (e.g. of objecs or walls). We show ha in general he POF is no bandlimied unless he surfaces are fla. We hen proide simple rules o esimae he essenial bandwidh of he POF for his model. Our analysis reeals ha, in addiion o he maximum and minimum dephs, he bandwidh of he POF also depend on he maximum surface slope and maximum frequency of pained signals. Wih a unifying formalism based on mulidimensional signal processing, we can erify seeral key resuls in POF processing, such as induced filering in space and deph correcion inerpolaion, and quanify he necessary sampling raes. Index Terms plenopic funcion, image-based rendering, sampling, specral analysis, bandwidh. I. INTRODUCTION Exising isual recording sysems use a single camera, and hus proide iewers wih a limied and passie iewing experience. The coninuing improemen in digial echnology has offered low-cos sensors and massie compuing power. This has led o he deelopmen of new sysems employing muliple cameras ogeher wih sophisicaed processing algorihms o delier unprecedened immersie recording and iewing capabiliies. Pracical sysems, called image-based rendering (IBR) [], ha synhesize arbirary irual iewpoins from seeral fixed sensors hae already emerged; see [2], [3], [4] for sureys of his area. A naural framework for sudying muliiew acquisiion and rendering is he concep of he plenopic funcion (POF) [5] ha describes he ligh inensiy passing hrough eery iewpoin, in eery direcion, for all ime, and for eery waelengh. The IBR problem can be reaed as an applicaion of he sampling heory o he POF. In his seing, acquired iews from he cameras proide discree samples of he POF, and he synhesized iew is reconsruced from he coninuous M. N. Do is wih he Deparmen of Elecrical and Compuer Engineering, he Coordinaed Science Laboraory, and he Beckman Insiue, Uniersiy of Illinois a Urbana-Champaign, Urbana IL 68 ( [email protected]). D. Marchand-Maille was a he Audioisual Communicaions Laboraory, École Polyechnique Fédérale de Lausanne (EPFL), CH-5 Lausanne, Swizerland. M. Veerli is wih he Audioisual Communicaions Laboraory, École Polyechnique Fédérale de Lausanne (EPFL), CH-5 Lausanne, Swizerland, and wih he Deparmen of Elecrical Engineering and Compuer Science, Uniersiy of California, Berkeley CA 9472 ( [email protected]). This work was suppored in par by he US Naional Science Foundaion under Gran CCR and he Swiss Naional Science Foundaion under Gran Plenus in Lain means complee or full. POF a a gien poin. The quesion of he minimum rae for sampling he POF can be addressed by specral analysis and esimaing he bandwidh of he POF. The firs sampling analysis for IBR was done by Chai e al. [6]. They analyzed he specral suppor of he POF o find an opimal uniform sampling rae for he POF. Zhang and Chen [7] exended IBR sampling for more general cases, including non-lamberian and occluded scenes. IBR sampling analysis is also reiewed in deail in a recen book [4]. In hese preious sudies, as for any specral-based echnique, he POF is assumed o be bandlimied. In his paper, we would like o examine more precisely he specral analysis and bandlimied assumpion of he POF. To faciliae his, we sudy a simple bu represenaie model where bandlimied signals (e.g. exure images) are pained on smooh surfaces (e.g. of objecs or walls). Using relaed mahemaical resuls on domain-warped bandlimied signals, we show ha, in general, he POF is no bandlimied unless he surface is fla. We hen proide simple rules o esimae he essenial bandwidh of he POF for his model. I is imporan o noe ha he POF is a powerful concepual ool for describing a number of problems in image/ideo processing, ision, and graphics. Mos acquired and synhesized forms of isual informaion, including images and ideos, can be reaed as low-dimensional slices (e.g. by fixing cerain ariables) of he POF. Hence, specral analysis of he POF has applicaions beyond IBR. For example, see [8] for an applicaion in ligh ranspor and [9], [] for applicaions in compuaional phoography. The ouline of he paper is as follows. In Secion II, we se up he scene and camera models and characerize he specral suppor of he POF. In Secion III, we sar focusing on he model in which bandlimied signals are pained on objec surfaces. In Secion IV, we discuss condiion for he POF o be bandlimied. In Secion V, we derie a simple rule o esimae he essenial bandwidh of ime-warped funcions. In Secion VI and Secion VII, we apply his rule o esimae he essenial bandwidhs of he POF and he sheared (or deph correced) POF and illusrae his wih some numerical experimens. Some preliminary resuls of his paper were presenend a a conference []. II. SCENE AND CAMERA MODELS A conenien way o parameerize he POF is o use he woplane parameerizaion, also known as ligh field or lumigraph [2], [3], as shown in Figure. By resricing he scene in a bounding box, each ligh ray can be specified by a pair of coordinaes (, u) and (, w) corresponding o he locaions of he camera and he image pixel wihin a camera, respeciely. Noe ha he image coordinae (, w) is defined relaiely wih

2 2 IEEE TRANSACTIONS ON IMAGE PROCESSING respec o he camera posiion (, u). Hence equialenly, (, u) specifies he iewing posiion and (u, w) specifies he iewing angle. The wo-plane parameerizaion fis he pinhole camera model [4], in which all pixels in a camera correspond o ligh rays ha are emied from one poin he camera posiion. The alue of he plenopic funcion p(, u,, w) is he ligh inensiy capured by a camera a locaion (, ) and a pixel locaion (, w) wihin ha camera. In general, p(, u,, w) is he ligh inensiy a he inersecion of he ray specified by (, u,, w) wih he neares objec surface o he camera posiion (, ). camera plane image plane objec surface z(x) θ Fig. 2. Scene model wih a funcional surface z(x). Coordinaes,, and z specify he camera posiion, pixel posiion, and deph, respeciely. The deph axis is rescaled so ha he focal lengh or he disance beween and axes is ; and hus he pixel posiion is relaed o he iewing angle θ by an(θ). s x u w Fig.. The wo-plane parameerizaion of he plenopic funcion. Each ligh ray is specified by a 4-D coordinae (, u,, w), where (, u) corresponds o he camera locaion in he camera plane and (, w) corresponds o he image poin (or pixel) in he image plane. Effeciely, (, u) specifies he iewing posiion and (u, w) specifies he iewing angle. For simpliciy of exposiion, and as in [6], [7], we consider a 2-D ersion of he POF, p(, ), by fixing u and w. This corresponds o he siuaion where he cameras are placed on a sraigh line and we consider he same image scan-line from each camera. Alernaiely, we could iew his as a flaland model where he 3D world is flaened ino a 2D plane. The funcion p(, ) is also known as epipolar-plane image (EPI) [5] and plays an imporan role in compuer ision [4], [6]. We consider he scene model, as shown in Figure 2, ha consiss of an objec surface (in he 2D seing of he POF, his a slice of he surface) specified by is arying deph z(). Wihou loss of generaliy, we rescale he deph alue z so ha he focal lengh or disance beween he camera and image planes is equal o. This scene model represens a microscale analysis of he plenopic funcion, where locally only one objec surface is isible. Suppose ha he ligh ray (, ) specified by he camera (or iewing) posiion and pixel posiion (or iewing angle) inersecs wih he objec surface a a poin wih coordinae (x, z(x)) as shown in Figure 2. Then simple geomeric relaions lead o x z(x) an(θ) x z(x). () Equaion () defines a fundamenal geomeric mapping ha links a ligh ray (, ) o a posiion s specified by x on he objec surface ha is seen by his ligh ray. We assume ha here is no self-occlusion on he objec surface in he field-of-iew of he cameras. This means ha each ligh ray (, ) wihin he field-of-iew can inersec wih a mos one poin on he objec surface z(x). This is equialen o requiring ha gien in () is a sricly monoonic funcion of x, which amouns o z (x) < max, (2) where he field-of-iew is limied by max. In oher words, he slope of he objec surface z(x) is bounded by he maximum iewing angle. Le l(x, ) be he ligh inensiy emied from he objec surface posiion x and iewing angle (see Figure 2). The funcion l(x, ) is also known as he surface ligh field [7] or surface plenopic funcion [7]. Then using () and under he no self-occlusion assumpion we hae p(, ) l(x, ), where x z(x). (3) Taking he Fourier ransform of he plenopic funcion p(, ) using (3) we obain P(ω, ω ) def F, {p(, )} p(, ) e j(ω+ω) dd l(x, ) e j(ω(x z(x))+ω) ( z (x)) dxd e jωx ( z (x)) l(x, ) e j(ω z(x)ω) ddx e jωx H(x, ω z(x)ω ) dx, (4) def where we denoe h(x, ) ( z (x)) l(x, ) and H(x, ω ) def F {h(x, )} h(x, ) e jω d. Similarly, we denoe L(x, ω ) def F {l(x, )}, and hen Fourier ransform properies lead o H(x, ω ) L(x, ω ) jz (x) L(x, ω ) ω. (5) Typically, excep for rare cases of highly specular surfaces, a a fixed surface posiion x he emied ligh inensiy l(x, ) changes ery slowly wih respec o he iewing angle. In he exreme case, he surface is ofen assumed o be Lamberian [6], which means l(x, ) l(x) for all. Thus, i is

3 DO, MARCHAND-MAILLET AND VETTERLI: ON BANDWIDTH OF PLENOPTIC FUNCTIONS 3 ω ω z max ω ω B L (a) ω z min ω ω ω max B L ω z max ω (b) ω z min ω ω ω 2π/ ω (c) Fig. 3. The specral suppors of he plenopic funcion p(, ). (a) The original suppor is conained beween wo lines corresponding o minimum and maximum dephs, plus an exended region accouning for non-lamberian surfaces. (b) Lowpass filering in he pixel dimension induces lowpass filering in he spaial dimension. (c) Sampling in space along leads o periodizaion in frequency along ω. reasonable o assume ha l(x, ) is a bandlimied funcion in he ariable. Using (4) and (5) we immediaely obain he following resul. Proposiion : Gien he no self-occlusion condiion (2) and suppose ha Then L(x, ω ), if ω > B L. (6) P(ω, ω ), if ω z(x)ω > B L for all x. (7) Therefore, as shown in Figure 3(a), he specral suppor of he plenopic funcion p(, ) is conained beween wo lines corresponding o minimum and maximum dephs, plus an exended region accouning for non-lamberian surfaces. This key finding was firs discoered by Chai e al. [6] for Lamberian surfaces and laer exended by Zhang and Chen [7] for non-lamberian surfaces. Howeer, in boh of hese preious works he deriaions are approximaions based on runcaing windows, in which he scene is approximaed by piece-wise consan deph segmens and he runcaion effec in he specral domain is ignored. Here, we show ha for noself-occlusion surfaces wih bandlimied ligh radiance, he resuling POF has specral suppor exacly conained in he region specified by (7). Moreoer, our analysis reeals he role of he surface slope z (x) as gien in he second erm in (5). This erm was ignored in preious analyses wih consan deph assumpion. This bow-ie shape specral suppor of he POF p(, ) makes i possible o induce coninuous-domain lowpass filering in he spaial dimension ia induced filering in he pixel dimension. Generally, i is physically impossible o realize coninuous-domain filering in he spaial dimension since we do no hae access o he POF in he coninuous domain of, bu raher only a discree locaions where we hae acual cameras. On he oher hand, coninuous-domain lowpass filering in he pixel dimension is possible by he opical sysem in he cameras. Because of he bow-ie shape specral suppor of he POF, Figure 3(b) illusraes ha lowpass filering in induces lowpass filering in as well. As a resul, Figure 3(c) shows ha we can sample he POF in space (i.e. by placing cameras a discree locaion along ) wihou alias. This induced filering propery also holds for sound signals as was shown in a sudy of he plenacousic funcion [8]. Typically, he plenopic funcion p(, ) is capured by cameras wih finie pixel resoluion along he pixel dimension. Thus, preious analyses [6], [7] assume ha P(ω, ω ) is bandlimied in he ω dimension o ω π/. Based on his assumpion and using Proposiion and Figure 3(b), i follows ha he bandwidh of he POF depends only on he range of dephs and he pixel resoluion. Howeer, he acual coninuous-domain plenopic funcion p(, ) migh no be bandlimied according o he camera resoluion. In some applicaions, i migh be of ineres o sudy he inrinsic bandwidh of he POF according o he underlying scene raher han he capuring deices. In his paper, we wan o characerize he bandwidh of he POF p(, ) according o a simple bu represenaie scene model ha will be described in he nex secion. III. SURFACE MODEL: SIGNALS PAINTED ON SURFACES Firs, we resric o Lamberian surfaces; i.e. l(x, ) l(x). Second, we assume ha he ligh radiance l(x) is resul of a bandlimied signal f(s) (e.g. exure image) pained on he objec surface, where s s(x) is he curilinear coordinae (i.e. s corresponds o he arc lengh) on he surface. Tha is l(x) f(s(x)). The surface coordinae x is deermined by he ligh ray coordinae (, ) as x x(, ) according o he geomeric mapping equaion () as x(, ) z(x(, )). (8) Wih a sligh abuse of noaion, we wrie s(, ) s(x(, )) for he composie mapping from ligh ray coordinae (, ) o he curilinear coordinae s on he surface. Wih hese mappings, we can relae he plenopic funcion p(, ) o he pained signal f(s) on he objec surface as p(, ) l(x(, )) f(s(x(, ))) f(s(, )). (9)

4 4 IEEE TRANSACTIONS ON IMAGE PROCESSING We sudy he bandwidh of he POF p(, ) by fixing eiher or. Noe ha fixing in he POF p(, ) corresponds o considering an image capured by a fixed camera, whereas fixing corresponds o considering signal recorded a a fixed pixel locaion by a moing camera. In boh cases, we obain a ime-warped funcion of a bandlimied funcion f(s()) where and s denoe a generic ariable and warping funcion, respeciely. Figure 4 depics his generic case sudy of he POF. f(s) pinhole camera projecion f(s()) Fig. 4. Mapping from f() o (f s)() f(s()) due o he pinhole camera projecion. Fixing eiher or and aking he deriaie of (8) wih respec o he oher ariable, we ge x(, ) z (x) () x(, ) z(x) z (x). () Hereafer, for breiy, in he righ-hand sides we wrie x for x(, ). The no-self-occlusion condiion (2) implies ha boh of hese parial deriaies are posiie for [ max, max ] or wihin he field-of-iew. This means ha x(, ) is a sricly monoonic funcion in each coordinae and. Using differenial relaion ds dx 2 + dz 2 + (z (x)) 2 dx, we obain he parial deriaies of s wih respec o and as s(, ) s(, ) ds x(, ) + (z (x)) 2 dx z (x) ds x(, ) z(x) + (z (x)) 2 dx z (x) (2) (3) From (2), we see ha if he surface is fla, i.e. z (x) is a consan hen s(, )/ is a consan or s(, ) is an affine funcion in for each fixed. Conersely, under he no-selfocclusion condiion (2), if s(, ) is affine for a fixed hen i is easy o see ha z (x) mus be a consan, and hus he surface mus be fla. Finally, we noe ha boh parial deriaies of s gien in (2) and (3) are greaer han. IV. BANDLIMITED PLENOPTIC FUNCTIONS As noed in he inroducion, o address he sampling problem of he plenopic funcion we need o sudy is specral suppor. In his secion, we examine he bandlimiedness of he plenopic funcion gien in (9). In plenopic sampling for IBR, he main ariable of ineres is, he camera posiion, as i leads o condiions on how o place he cameras. So le us consider he siuaion where he pixel posiion is fixed, and for breiy we drop he ariable in funcions in his secion. Again, suppose ha he pained signal f(s) is bandlimied. From he discussion a he end of he las secion we noe ha if he surface in our scene is fla, hen s() is affine and he plenopic funcion p() f(s()) is a uniformly sreched ersion of f(). Thus, i follows immediaely from he shifing and scaling properies of he Fourier ransform ha f(s()) is also bandlimied. We are ineresed o know if here are any oher surfaces ha resul in bandlimied plenopic funcions. Time-warped bandlimied funcions hae been sudied in he signal processing lieraure. In [9], Clark conjecured ha when a bandlimied funcion f is warped by a monoonic funcion s, he resuling funcion (f s)() f(s()) is also bandlimied if and only if s() is affine. In [2], his conjecure was proed for a large class of s(), in paricular for s() ha on cerain ineral is a resricion of an enire funcion. 2 Laer, in [2], Clark s conjecure was shown o be false by a peculiar counerexample consruced by Y. Meyer. Howeer, ha paper also noed ha i is no possible for a non-affine warping funcion o presere bandlimiedness in general. Unaware of his line of work, in [22], we made he same conjecure on he preseraion of bandlimiedness under warping. The implicaion of he aboe resul is ha in general, he plenopic funcion is no bandlimied unless he surface is fla. In he nex secions we will sudy he essenial bandwidh, defined as he bandwidh where mos of he signal energy resides, of he plenopic funcion for general smooh surfaces. V. BANDWIDTH OF TIME-WARPED FUNCTIONS Le denoe g() (f s)() o be a ime-warped funcion ha models he plenopic funcion as was described in Secion III. Is Fourier ransform is G(ξ) f(s())e jξ d. (4) Le F(ω) be he Fourier ransform of f(s). Then, f(s) 2π F(ω)e jωs dω. (5) Subsiuing (5) ino (4) we obain G(ξ) ( ) F(ω) e jωs() e jξ d dω 2π F(ω)K s (ξ, ω)dω, (6) 2π where K s (ξ, ω) def F {e jωs() } is he Fourier ransform of e jωs(). The kernel funcion K s (ξ, ω) characerizes how he warping funcion s broadens he specrum of f in he warped funcion g f s. To see his effec, firs consider he case when s is an affine funcion: s() a + b. In his case we hae K s (ξ, ω) F {e jω(a+b) } 2πe jaω δ(ξ bω), (7) 2 An enire funcion is a funcion of complex ariable ha has deriaie a each poin in he enire finie plane. In paricular, bandlimied funcions are enire funcions.

5 DO, MARCHAND-MAILLET AND VETTERLI: ON BANDWIDTH OF PLENOPTIC FUNCTIONS 5 which is concenraed along he line ω ξ/b. Subsiue (7) ino (6) we ge ( ) G(ξ) ejaξ/b ξ F. b b Thus, for he affine warping s() a + b we can relae he bandwidh of he warped funcion g f s o he bandwidh of f as BW g a BW f s BW f. (8) Nex, consider a more general siuaion in which he warping funcion s deiaes from an affine funcion as Then he kernel K s (ξ, ω) becomes s() a + b + s(). (9) K s (ξ, ω) F {e jω(a+b) e jω s() } 2πe jaω δ(ξ bω) ξ F {e jω s() }. (2) Consider a simple case where he deiaion s() is an oscillaion funcion wih a single frequency µ >, i.e. s() c sin(µ). Using he following expansion e jx sin(α) n J n (x)e jnα, where J n (x) is he n-h order Bessel funcion of he firs kind, we hae F {e jωc sin(µ) } 2π J n (cω) δ(ξ nµ). n The Bessel funcions J n (x) decay exponenially for sufficienly large n and are negligible for n > x +. Thus, for s() c sin(µ), he Fourier ransform F {e jω s() } is essenially zero for frequency ξ > cµω + µ. Based on his approximaion, subsiuing back in (2) and hen (6), we see ha he essenial bandwidh of g is essbw g ( b + cµ ) BW f + µ. (2) Noe ha in his case s() a+b+c sin(µ), we can wrie b + cµ max s. Moreoer, for he plenopic funcion, ypically he oscillaion of he surface s is much smaller han he oscillaion of he pained exure f, herefore µ BW f. Furhermore, as noed a he end of Secion III, for plenopic funcion s >. Thus, from (2) we can wrie essbw g (max s ) BW f. (22) In words, he essenial bandwidh of he ime-warped funcion g() f(s()) can be approximaed by he produc of he maximum deriaie of s wih he bandwidh of f. The rule (22) can be approximaed for general warping funcion s as follows. Since ypically s is smooh, i can be approximaed by a piecewise linear funcion ŝ() def s( k ) + s (ξ k )( k ) for [ k, k+ ] (23) wih appropriaely chosen ξ k [ k, k+ ] and sufficienly small segmens [ k, k+ ]. Then he ime-warped funcion g() f(s()) can be approximaed by ĝ() def f(ŝ()) (24) f(s( k ) + s (ξ k )( k )) for [ k, k+ ) k f(s( k ) + s (ξ k )( k )) b k (), (25) where b k () is he indicaor funcion of he ineral [ k, k+ ]; i.e. b k () equals o for [ k, k+ ] and oherwise. Denoe f k () f(s( k )+s (ξ k )( k )), which is an affine warping. Then we can relae he bandwidh of f k o he bandwidh of f as BW fk s (ξ k ) BW f. Since b k () is a recangular funcion of lengh ( k+ k ), is Fourier ransform B k (ω) is a sinc funcion wih essenial bandwidh essbw bk. k+ k From (25) i follows ha Ĝ(ω) k F k(ω) B k (ω), and hence ( ) essbwĝ max s (ξ k ) BW f +. (26) k k+ k For he plenopic funcion, ypically he oscillaion of s is much smaller han he oscillaion of f. Therefore, a good approximaion of f(s()) can be obained from f(ŝ()) using a piecewise linear approximaion of s() as in (23) wih max k /( k+ k ) BW f. Thus, we can discard he second erm on he righ-hand side of (26) and obain essbw g essbwĝ (max s ) BW f, which is he desired resul (22). Noe ha his approximaion is exac for affine warping s as shown in (8). The bandwidh analysis of ime-warped funcions in his secion follows he bandwidh analysis of FM (frequencymodulaion) signals in communicaion sysems [23]. A similar rule like (22) is called he Carson s rule in he communicaion sysems lieraure. We noe ha boh he Carson s rule and (22) are difficul o proe precisely, excep for some paricular cases, and hus hey should be iewed as rules of humb. In he nex secions, we will show ha he rule (22) is quie accurae and proides an effecie mean o esimae he bandwidh of plenopic funcions. VI. BANDWIDTH OF PLENOPTIC FUNCTIONS The resul (22) from he las secion reeals he role of he maximum absolue alue of he deriaies of he curilinear coordinae s on he bandwidh of he plenopic funcion. These maximum deriaies represen he wors cases of he muliplicaie erm in he bandwidh expansion of he POF as gien in (22). From (2) and (3) we hae max max s(, ) s(, ) + max z 2 max max z (27) z max + max z 2 max max z. (28)

6 6 IEEE TRANSACTIONS ON IMAGE PROCESSING Wih hese maximum deriaies, ogeher wih he knowledge of he bandwidh of he pained signal (or exure image) f on he objec surface, we obain essenial bandwidh esimaes for he POF using (22). Inuiiely, he resuls (27)- (28) imply ha when arying camera posiion, he worse case of bandwidh expansion comes from he seepes slope (wih respec o he camera axis) on he objec surface. When arying pixel posiion, he wors case happens when he surface is a he seepes poin and furhes, and he pixel is a he boundary of he field-of-iew. These findings are also noed in he lieraure on exure mapping and image warping [24]. Fig. 5. objec surface z C α R θ camera s ligh ray The scene wih a cured wall ha is used in numerical experimens. To illusrae and alidae hese rules for esimaing he bandwidh of he POF we consider a synheic scene as shown in Figure 5. Noe ha as before, all lengh measures are normalized so ha he focal lengh (i.e. he disance beween and axes) is equal o. In he scene, here is a cured wall as an arc of a circle wih radius R and cener a disance C from he origin on he z axis. A exure signal f(s) sin(2πκs) of frequency κ is pained on he wall. For conenience, we also specify a poin on he objec surface by he angle α beween he corresponding radial line wih he z-axis (see Figure 5). Then x R sin(α) C z(x) cos(α) (29) R s αr. I follows ha z (x) an(α). Subsiuing (29) ino he geomeric mapping equaion () and using θ o specify he pixel posiion, an(θ), we ge R sin(θ) (C R cos(θ))an(θ). From his we can express he curilinear coordinae s on he objec surface hrough he ligh ray coordinaes (, ) as (noing θ an ()) s R ( sin ( cos(θ) + C sin(θ) R ) ) θ. (3) Figure 6(b) shows he resuling POF of a cured wall wih C 2, R, and κ 2. The examined ranges of and are: [ 3, +3], and [.35, +.35] (wih he focal lengh normalized o, his is equialen o using 5 mm lens on a 35 mm camera). Wih hese parameers, he surface deph is in he range z(x) [, 3.76] and he surface slope is in he range z (x) [.25,.25]. Plugging hese alues ino (27)-(28) and (22), we obain he following esimaes for he essenial maximum frequencies of P(ω, ω ) as { ω max /(2π) 5.7Hz, ω max /(2π) 78.6Hz. These esimaes agree well wih he plo of P(ω, ω ) for cured wall in Figure 6(d). Also noe in he plo ha he specral suppor of he POF is sandwiched beween he lines ω z min ω and ω z max ω as illusraed in Figure 3. For comparison, Figure 6(a) and Figure 6(c) show he POF and is specrum for he same camera configuraion and pained exure, excep he wall is fla a consan deph z(x).5. Comparing o he fla wall case, he specrum suppor of he POF of he cured wall is significanly broader, boh in he angle and he radial lengh of he cone-shape region illusraed in Figure 3. The angular broadening of he POF specrum is due o he arying of surface deph z(x), as was noed preiously in [6]. Howeer he radial broadening of POF specrum, which is due o he maximum surface slope z (x) as indicaed in (27)-(28), is only reealed in his work. VII. BANDWIDTH OF SHEARED PLENOPTIC FUNCTIONS The resuls in he las secion characerize he bandwidhs of he plenopic funcion p(, ) in each dimension and separaely. For plenopic sampling, such bandwidhs are relean if we sample and reconsruc along each dimension and separaely while fixing he oher dimension. Howeer, he ypical shapes of POF specral suppors as seen in Figure 3 and Figure 6(c-d) indicae ha we can compac he POF specrum more (and hence hae less aliasing in sampling) by nonseparably process he wo dimensions and. In paricular, using he knowledge of minimum and maximum dephs, z min and z max, and he propery of POF specral suppor as shown in Figure 3, opimal non-separable reconsrucion filers for IBR were deried in [6], [7]. An alernaie approach o explore his propery of he POF specrum suppor in IBR using only D reconsrucion filer is as follows. Since he POF specral suppor is slaned according o he deph range, shearing he POF specrum as shown from Figure 7(c) o Figure 7(d) would make i more compac along ω axis. Figure 7(a) and Figure 7(b) illusrae he spaial suppors of he corresponding funcions wih specra gien in Figure 7(c) and Figure 7(d); namely, he POF and is sheared ersion. More precisely, he desired shearing operaor is obained by he following change of ariable in he frequency domain { ω ω ω /z, ω ω.

7 DO, MARCHAND-MAILLET AND VETTERLI: ON BANDWIDTH OF PLENOPTIC FUNCTIONS (a) Fla wall: p(, ) (b) Cured wall: p(, ) ω /(2π).2 ω /(2π) ω /(2π) ω /(2π) (c) Fla wall: P(ω, ω ) (d) Cured wall: P(ω, ω ) Fig. 6. Examples of he plenopic funcion p(, ) and is Fourier ransform P(ω, ω ) for fla and cured walls. The corresponding change of ariable in he space domain is {, + /z. Geomerically, his shearing operaor maps, in he frequency domain, he line ω z ω ino he ω axis [i.e. from Figure 7(c) o Figure 7(d)]; or equialenly, in he space domain, i maps he axis ino he line /z [i.e. from Figure 7(a) o Figure 7(b)]. Therefore, wih a suiable choice of z such ha z min z z max, he specrum of sheared POF is more compac along he ω axis. The opimal deph z suggesed in [6] saisfies ( + ), z 2 z min z max which can be obained hrough Figure 3. Wih he compac specrum along he w axis we can achiee high qualiy reconsrucion (i.e. less aliasing) for he sheared POF by simply reconsrucing along axis for each fixed. In oher words, we inerpolae he sheared POF along lines. I is easy o see ha corresponding o he line in he sheared domain is he following line in he original domain [see Figure 7(a-b)] + /z + /z. (3) Therefore, equialenly, we can obain high qualiy reconsrucion of he original plenopic funcion p(, ) a a locaion (, ) by inerpolaing along he line (3). From (), we see ha all corresponding ligh rays (, ) ha saisfy he line equaion (3) inersec wih he ligh ray (, ) a a same poin of deph z. The Lumigraph [3] sysem employs he same reconsrucion sraegy which hey call deph correcion inerpolaion. The auhors of [3] refer o he line (3) as an opical flow line (where he objec surface seen by he ligh ray (, ) is assumed o be a deph z ), and hey expec he plenopic funcion o be smooh along he opical flow lines. Their experimens show ha reconsrucion by deph-correced inerpolaion along he opical flow lines has significanly higher qualiy compared o uncorreced inerpolaion (i.e. inerpolae along same pixel lines ). We can characerize he smoohness of he plenopic funcion along he opical flow line (3) by esimaing he bandwidh of he D slice funcion of he POF along his line (which is also he D funcion along he line of he sheared POF). The corresponding deriaie for he bandwidh expanding facor in (22) is he direcional deriaie of s along he line (3) wih he uni ecor u (, /z ). Using (2)- (3), we obain he deriaie of s in his direcion as D u s(, ) u s(, ) + u s(, ) ( z(x)/z ) + (z (x)) 2 z. (32) (x)

8 8 IEEE TRANSACTIONS ON IMAGE PROCESSING + /z + /z /z (, ) (a) Plenopic funcion (b) Sheared plenopic funcion ω ω ω z ω ω ω (c) Specrum of POF (d) Specrum of sheared POF Fig. 7. Sheared specrum of he plenopic funcion by a change of ariable: ω ω (/z )ω, ω ω. Wih approriae choice of z, he sheared specrum is mos compaced near he ω axis. Comparing D u s in (32) o s/ in (2), we see ha, wih a suiable choice of z such as z (z min + z max )/2, he absolue alue of he deriaie of s in he direcion u (, /z ) is smaller han he one in he direcion (, ). Hence, according o (22), he bandwidh of he POF along he opical flow line (3) is smaller han he bandwidh along he same pixel line. Figure 8 shows example slices of he POF for he cured wall scene described in Secion VI along he same pixel line and opical flow line (3) wih and z.5. We see ha he maximum frequency of he POF slice along he opical flow is much smaller compared o he one along he same pixel line, which confirms he adanage of deph-correced inerpolaion in IBR. Using (2) and (32), and he surface deph and slope ranges for cured wall scene found in Secion VI, we obain esimaes for he maximum frequency for hese wo slices of he POF as 3.2 Hz and.6 Hz, respeciely. These esimaes closely characerize he funcion plos in Figure 8. surface is fla. We hen derie a simple rule o esimae he essenial bandwidh, defined as he bandwidh where mos of he signal energy resides, of he plenopic funcion for his model. This essenial bandwidh is esimaed as he produc of he bandwidh of he pained signal imes he maximum absolue deriaie of he surface curilinear coordinae along a cerain direcion. Our analysis reeals ha, in addiion o he maximum and minimum surface dephs, he bandwidhs of he plenopic funcion also depend on he maximum surface slope and maximum frequency of he pained signal. By reaing he POF wih a unifying formalism based on mulidimensional signal processing, we can erify seeral imporan resuls, including induced filering along he camera dimensions, deph correcion inerpolaion in Lumigraph, and quanifying he necessary sampling raes. Numerical resuls show ha he resuling esimaed bandwidhs of plenopic funcions are accurae and effeciely characerize he performance of image based rendering algorihms. VIII. CONCLUSION In his paper we sudied he bandwidh of he plenopic funcion of a simple scene model where a bandlimied signal is pained on a smooh surface. We show ha in general he plenopic funcion for his model is no bandlimied unless he Acknowledgmens. The lae Daid Slepian (923-27) and his seminal paper [25] hae been inspiraions for us in compleing his paper. We also hank Niin Aggarwal (UIUC) for he helpful discussions on he bandwidh of ime-warped funcions.

9 DO, MARCHAND-MAILLET AND VETTERLI: ON BANDWIDTH OF PLENOPTIC FUNCTIONS 9 p(, /z ) p(,) (a) [7] G. Miller, S. Rubin, and D. Ponceleon, Lazy decompression of surface ligh fields for precompued global illuminaion, in Eurographics Rendering Workshop, Vienna, Ausria, 998, pp [8] T. Ajdler, L. Sbaiz, and M. Veerli, The plenacousic funcion and is sampling, IEEE Trans. Signal Proc., ol. 54, no., pp , 26. [9] D. Cochran and J. J. Clark, On he sampling and reconsrucion of ime warped band-limied signals, in Proc. IEEE In. Conf. Acous., Speech, and Signal Proc., Apr. 99. [2] X.-G. Xia and Z. Zhang, On a conjecure on ime-warped band-limied signals, IEEE Trans. Signal Proc., ol. 4, pp , Jan [2] S. Azizi, D. Cochran, and J. N. McDonald, On he preseraion of bandlimiedness under non-affine ime warping, in Proc. In. Workshop on Sampling Theory and Applicaions, Aug [22] D. Marchand-Maille, Sampling heory for image-based rendering, Maser s hesis, Swiss Federal Insiue of Technology Lausanne, 2. [23] B. P. Lahi, Modern Digial and Analog Communicaion Sysems, 3rd ed. Oxford Uniersiy Press, 998. [24] G. Wolberg, Digial Image Warping. IEEE Compuer Sociey Press, 994. [25] D. Slepian, On bandwidh, Proc. IEEE, ol. 64, no. 3, pp , Mar (b) Fig. 8. Slices of he plenopic funcion of he cured wall scene along: (a) same pixel line. (b) opical flow line wih z.5. REFERENCES [] L. McMillan and G. Bishop, Plenopic modeling: an image-based rendering sysem, in Proc. SIGGRAPH, 995, pp [2] H.-Y. Shum, S. Kang, and S.-C. Chan, Surey of image-based represenaions and compression echniques, IEEE Trans. Circ. and Sys. for Video Tech., ol. 3, pp. 2 37, No. 23. [3] C. Zhang and T. Chen, A surey on image-based rendering represenaion, sampling and compression, EURASIP Signal Proc.: Image Commun., ol. 9, pp. 28, Jan. 24. [4] H.-Y. Shum, S.-C. Chan, and S. B. Kang, Image-Based Rendering. Springer, 27. [5] E. H. Adelson and J. R. Bergen, The plenopic funcion and he elemens of early ision, in Compuaional Models of Visual Processing, M. Landy and J. A. Moshon, Eds. MIT Press, 99, pp [6] J.-X. Chai, X. Tong, S.-C. Chan, and H.-Y. Shum, Plenopic sampling, in Proc. SIGGRAPH, 2, pp [7] C. Zhang and T. Chen, Specral analysis for sampling image-based rendering daa, IEEE Trans. Circ. and Sys. for Video Tech., ol. 3, pp. 38 5, No. 23. [8] F. Durand, N. Holzschuch, C. Soler, E. Chan, and F. X. Sillion, A frequency analysis of ligh ranspor, in Proc. SIGGRAPH, 25, pp [9] R. Ng, Fourier slice phoography, in Proc. SIGGRAPH, 25, pp [] A. Veeraraghaan, R. Raskar, A. Agrawal, A. Mohan, and J. Tumblin, Dappled phoography: Mask enhanced cameras for heerodyned ligh fields and coded aperure refocusing, in Proc. SIGGRAPH, 27. [] M. N. Do, D. Marchand-Maille, and M. Veerli, On he bandlimiedness of he plenopic funcion, in Proc. IEEE In. Conf. on Image Proc., Genoa, Ialy, Sep. 25. [2] M. Leoy and P. Hanrahan, Ligh field rendering, in Proc. SIGGRAPH, 996, pp [3] S. Gorler, R. Grzeszczuk, R. Szeliski, and M. Cohen, The lumigraph, in Proc. SIGGRAPH, 996, pp [4] O. D. Faugeras, Three-Dimensional Compuer Vision: A Geomeric Viewpoin. MIT Press, 993. [5] R. C. Bolles, H. H. Baker, and D. H. Marimon, Epipolar-plane image analysis: an approach o deermining srucure from moion, Inernaional Journal of Compuer Vision, ol., no. 7, pp. 7 55, 987. [6] D. A. Forsyh and J. Ponce, Compuer Vision: A Modern Approach. Prenice-Hall, 22.