Three-Dimensional Scene Flow

Transcription

1 Appeared n the 7th Internatonal Conference on Computer Vson, Corfu, Greece, September Three-Dmensonal Scene Flow Sundar Vedula y, Smon Baker y, Peter Rander yz, Robert Collns y, and Takeo Kanade y y The Robotcs Insttute, Carnege Mellon Unversty, Pttsburgh, PA z Zaxel Systems Inc., Ten 40th Street, Pttsburgh, PA 152 Abstract Scene flow s the three-dmensonal moton feld of ponts n the world, just as optcal flow s the twodmensonal moton feld of ponts n an mage. Any optcal flow s smply the projecton of the scene flow onto the mage plane of a camera. In ths paper, we present a framework for the computaton of dense, non-rgd scene flow from optcal flow. Our approach leads to straghtforward lnear algorthms and a classfcaton of the task nto three major scenaros: (1) complete nstantaneous knowledge of the scene structure, (2) knowledge only of correspondence nformaton, and (3) no knowledge of the scene structure. We also show that multple estmates of the normal flow cannot be used to estmate dense scene flow drectly wthout some form of smoothng or regularzaton. 1 Introducton Optcal flow s a two-dmensonal moton feld n the mage plane. It s the projecton of the three-dmensonal moton of the world. If the world s completely non-rgd, the motons of the ponts n the scene may all be ndependent of each other. One representaton of the scene moton s therefore a dense three-dmensonal vector feld defned for every pont on every surface n the scene. By analogy wth optcal flow, we refer to ths three-dmensonal moton feld as scene flow. In ths paper, we present a framework for the computaton of dense, non-rgd scene flow drectly from optcal flow. Our approach leads to effcent lnear algorthms and a classfcaton of the task nto three major scenaros: 1. Complete nstantaneous knowledge of the structure of the scene, ncludng surface normals and rates of change of depth maps. In ths case, only one optcal flow s requred to compute the scene flow. 2. Knowledge only of stereo correspondences. In ths case, at least two optcal flows are needed to compute the scene flow, but more mprove robustness. 3. No knowledge of the surface. In ths case, several optcal flows can be used n a reconstructon algorthm to estmate the scene structure (and then scene flow). For each scenaro, we propose an algorthm and demonstrate t on a collecton of vdeo sequences of a dynamc, non-rgd scene. We also show that multple estmates of the normal flow cannot be used to estmate scene flow drectly, wthout some form of regularzaton or smoothng. One possble applcaton of scene flow s as a predctor for effcent and robust stereo. Gven a reconstructed model of the scene at a certan tme, one would lke to obtan an estmate of the structure at the next tme step usng mnmal computaton. Ths would allow: (1) more effcent computaton of the structure at the next tme step because a frst estmate would be avalable to reduce the search space, and (2) more robust computaton of the structure because the predcted structure can be ntegrated wth the new stereo data. Other applcatons of scene flow nclude varous dynamc renderng and nterpretaton tasks, from the generaton of slow-moton replays, to the understandng and modelng of human actons. 1.1 Related Work Computng the three-dmensonal moton of a scene s a fundamental task n computer vson that has been approached n a wde varety of ways. If the scene s rgd and the cameras are calbrated, the three-dmensonal scene structure and relatve moton can be computed (up to a scale factor) from a sngle monocular vdeo sequence usng structure-from-moton [Ullman, 1979]. If the scene s only pecewse rgd, extensons to structure-from-moton algorthms can be used. See, for example, [Zhang and Faugeras, 1992a] and [Costera and Kanade, 1998]. Although restrcted forms of non-rgdty can be analyzed usng the structure-from-moton paradgm [Avdan and Shashua, 1998], general non-rgd moton cannot be estmated from a sngle camera wthout addtonal assumptons about the scene. However, gven strong enough a pror assumptons about the scene, for example n the form of a deformable model [Pentland and Horowtz, 1991] [Metaxas and Terzopoulos, 1993] or the assumpton that the moton mnmzes the devaton from a rgd body moton [Ullman, 1984], recovery of three-dmensonal nonrgd moton from a monocular vew s possble. See [Penna, 1994] for a recent survey of monocular non-rgd 722

2 moton estmaton, and the assumptons used to compute t. Another common approach to recoverng threedmensonal moton s to use multple cameras and combne stereo and moton n an approach known as motonstereo. Nearly all moton-stereo algorthms assume that the scene s rgd. See, for example, [Waxman and Duncan, 1986], [Young and Chellappa, 1999], and[zhang and Faugeras, 1992b]. A paper whch explctly combnes two optcal flow felds s that of [Sh et al., 1994]. In ths paper, both the analyss and mplementaton are only applcable to certan smple motons of the camera (.e. translatons). A few moton-stereo papers do consder non-rgd moton, ncludng [Lao et al., 1997] and [Malassots and Strntzs, 1997]. The former uses a relaxaton-based algorthm to co-operatvely match features n both the temporal and spatal domans. It therefore does not provde dense moton. The latter uses a grd whch acts as a deformable model n a generalzaton of the monocular approaches mentoned above. Besdes requrng apror models of the scene, most deformable-model based approaches to moton-stereo would be too neffcent for our stereo-predcton applcaton. 2 Image Formaton Prelmnares Consder a non-rgdly movng surface f(x; y; z; t) =0 maged by a fxed camera, wth 3 4 projecton matrx P, as llustrated n Fgure 1. There are two aspects to the formaton of the mage sequence I = I (u ;v ; t) captured by camera : (1) the relatve camera and surface geometry, and (2) the llumnaton and surface photometrcs. 2.1 Relatve Camera and Surface Geometry The relatonshp between a pont (x; y; z) on the surface and ts mage coordnates (u ;v ) n camera s gven by: u = [P ] 1 (x; y; z; 1) T [P ] 3 (x; y; z; 1) T (1) v = [P ] 2 (x; y; z; 1) T [P ] 3 (x; y; z; 1) T (2) where [P ] j s the j th row of P. Equatons (1) and (2) descrbe the mappng from a pont x = (x; y; z) on the surface to ts mage u = (u ;v ) n camera. Wthout knowledge of the surface, these equatons are not nvertble. Gven f, they can be nverted, but the nverson requres ntersectng a ray n space wth the surface f. The dfferental relatonshps between x and u can be represented by a 2 3 Jacoban The 3 columns of the Jacoban matrx store the dfferental change n projected mage co-ordnates per unt change n x, y, andz. A closed-form expresson as a functon of x can be derved by dfferentatng Equatons (1) and (2) symbolcally. The descrbes the relatonshp between a small change n the pont on the surface and ts mage n z y Surface f x (x,y,z) n Camera P v v δ δ x δu δx u Illumnaton Flux E (Irradance) Image I (u,v ) Center of Projecton Fgure 1: A non-rgd surface f (x; y; z; t) =0s movng wth respect to a fxed world coordnate system (x; y; z). The normal to the surface s n = n(x; y; z; t). The surface s assumed to be Lambertan wth albedo = (x; y; z; t) and the llumnaton flux (rradance) s E. The th camera s fxed n space, has a coordnate frame (u;v), s represented by the 34 camera matrx P, and captures the mage sequence I = I(u;v; t). camera va x. Smlarly, the nverse Jacoban descrbes the relatonshp between a small n a pont n the mage of camera and the pont t s magng n the scene va x Snce mage co-ordnates do not map unquely to scene co-ordnates, the nverse Jacoban cannot be computed wthout knowledge of the surface. If we know the surface (and ts gradent), the nverse Jacoban can be estmated as the soluton of the followng two sets of lnear = (3) = = (0 0): (4) Equaton (3) expresses the constrant that a small change n u must lead to a small change n x whch when projected back nto the mage gves the orgnal change n u. Equaton (4) expresses the constrant that a small change n u does not lead to a change n f snce the correspondng pont n the world should stll le on the surface. The 6 lnear equatons n Equatons (3) and (4) can be decoupled nto 3 exst for Unque solutons f and only f: rf 6= 0: (5) Snce rf s parallel to the surface normal n, the equatons are degenerate f and only f the ray jonng the camera center of projectonand x s tangent to the surface.

3 2.2 Illumnaton and Surface Photometrcs At a pont x n the scene, the rradance or llumnaton flux measured n the drecton m at tme t can be represented by E = E(m; x; t) [Horn, 1986]. Ths 6D rradance functon E s what s descrbed as the plenoptc functon n [Adelson and Bergen, 1991]. We denote the net drectonal rradance of lght at the pont (x; y; z) on the surface at tme t by s = s(x; y; z; t). The net drectonal rradance s s a vector quantty and s gven by the (vector) surface ntegral of the rradance E over the vsble hemsphere of possble drectons: s(x; y; z; t) = Z S(n) E(m; x; y; z; t)dm (6) where S(n) =fm : kmk = 1 and mn 0g s the hemsphere of drectons from whch lght can fall on a surface patch wth surface normal n. We assume that the surface s Lambertan wth albedo = (x; t). Then, assumng that the pont x =(x; y; z) s vsble n the th camera, and that the ntensty regstered n mage I s proportonal to the radance of the pont that t s the mage of (.e. mage rradance s proportonal to scene radance [Horn, 1986]), we have: I (u ; t) =,C (x; t)[n(x; t) s(x; t)] (7) where C s a constant that only depends upon the dameter of the lens and the dstance between the lens and the mage plane. The mage pxel u =(u ;v ) and the surface pont x =(x; y; z) are related by Equatons (1) and (2). 3 Two-Dmensonal Optcal Flow Suppose x(t) s the 3D path of a pont on the surface and the mage of ths pont n camera s u (t). The 3D moton of ths pont s and the 2D mage moton of ts projecton s du du. The 2D flow feld s usually known as optcal flow. As the pont x(t) moves on the surface, t s natural to assume that ts albedo = (x(t); t) remans constant;.e. we assume that d = 0: (8) (For a deformably movng surface, t s only the surface propertes lke albedo that dstngush ponts anyway). The bass for optcal flow algorthms s then the equaton: di = ri du =,C (x; t) d [n s] (9) where ri s the spatal gradent of the mage, du s the optcal flow, s the nstantaneous rate of change of the mage ntensty I = I (u ; t). The term ns depends upon both the shape of the surface (n) and the llumnaton (s). To avod explct dependence upon the structure of the three-dmensonal scene, t s often assumed that: n s = Z S(n) E(m; x; t) n dm (10) s constant ( d [n s] =0). Wth unform llumnaton or a surface normal that does not change rapdly, ths assumpton holds well (at least for Lambertan surfaces). In ether scenaro di goes to zero, and we arrve at the Normal Flow or Gradent Constrant Equaton, used by dfferental optcal flow algorthms [Barron et al., 1994]: ri du = 0: (11) Usng ths constrant, a large number of algorthms have been proposed for estmatng the optcal flow du. See [Barronet al., 1994] for a recent survey. 4 Three-Dmensonal Scene Flow In the same way that optcal flow descrbes an nstantaneous moton feld n an mage, we can thnk of scene flow as a three-dmensonal flow feld descrbng the moton at every pont n the scene. The analyss n Secton 2.1 was only for a fxed tme t. Now suppose there s a pont x = x(t) movng n the scene. The mage of ths pont n camera s u = u (t). If the camera s not movng, the rate of change of u s unquely determned as: du : (12) Invertng ths relatonshp s, agan, mpossble wthout knowledge of the surface f. To nvert t, note that x depends not only on u, but also on the tme, ndrectly through the surface f = f(x; t). Thatsx = x(u (t); t). Dfferentatng ths expresson wth respect to tme gves: du + u : (13) Ths equaton says that the moton of a pont n the world s made up of two components. The frst s the projecton of the scene flow on the plane tangent to the surface and passng through x. Ths s obtaned by takng the nstantaneous moton on the mage plane (the optcal flow du ), and projectng t out nto the scene usng the nverse The second term s the contrbuton to scene flow arsng from the three-dmensonal moton of the pont n the scene maged by a fxed pxel. It s the nstantaneous moton of x along the ray correspondng to u. The magntude of s (proportonal to) the rate of change of the depth u of the surface f along ths ray. A dervaton of u s presented n Appen A. There are three major ways of computng scene flow, dependng upon what s known about the scene at that nstant:

4 t =1 t =2 t =3 t =4 t =5 Fgure 2: A sequence of mages that show the scene moton. For lack of space, we only present scene flow results for t =1n ths paper. The extended sequence s presented to help the reader vsualze the three-dmensonal moton. 1. Completely known nstantaneous structure of the scene, ncludng surface normals, depth maps, and the temporal rate of change of these depth maps. 2. Knowledge only of stereo correspondences. Snce we are workng n a calbrated settng, ths s equvalent to havng the depth maps. However, t does not nclude the surface normals and the temporal rates of change of the depth maps. 3. Completely unknown scene structure. We do not even know correspondence nformaton. Each of these cases leads to a dfferent strategy for estmatng the scene flow. It seems ntutve that less knowledge of scene structure requres the use of more optcal flows, and ndeed ths result does follow from the amount of degeneracy n the lnear equatons used to compute scene flow. We now descrbe algorthms for each of the three cases. We also demonstrate ther valdty usng flow results computed from multple mage sequences (captured from varous vewponts) of a non-rgd, dynamcally changng scene. One such mage sequence s shown n Fgure Complete Knowledge of Surface Geometry If the surface f s completely known (wth hgh accuracy), the surface gradent rf can be computed at every pont. The nverse Jacoban can then be estmated solvng the set of 6 lnear equatons n Equatons (3) and (4). Gven the nverse Jacoban, the scene flow can be estmated from the optcal flow du usng Equaton (13): du + u : (14) Computng requres the temporal dervatve of the u surface depth map, and s descrbed n Appen A. Complete knowledge of the scene structure thus enables us to compute scene flow from one optcal flow, (and the rate of change of the depth map correspondng to ths mage.) These two peces of nformaton correspond to the two components of the scene flow; the optcal flow s projected onto the tangent plane passng through x, anhe rate of change of depth map s mapped onto a component along the ray passng through the scene pont x and the center of projecton of the camera. Fgure 3: The horzontal and vertcal optcal flows at t =1for the same camera used n Fgure 2. Darker pxels ndcate moton to the left and top of the frames respectvely. Note that we assume that the surface s locally planar when computng the nverse Jacoban. Snce the surface s known, t s possble to project the flowed pont n the mage and ntersect ths ray wth the surface. We currently do not perform ths to save an expensve ray-surface ntersecton for every pxel. If only one optcal flow s used, the scene flow can be computed only for those ponts n the scene that are vsble n that mage. It s possble to use multple optcal flows n multple cameras for better vsblty, and for greater robustness. Also, flow s recovered only when the change n depth map s vald - that s, when an ndvdual pxel sees neghborng parts of the surface as tme changes. If the moton of a surface s large relatve to ts sze, then a pxel vews dfferent surfaces, and flow cannot be computed. Consder the scene shown as a sequence n Fgure 2. For lack of space, we only present scene flow results for tme t =1n ths paper. The scene flow s computed usng depth maps from the model obtaned from the volumetrc mergng of multplerange mages, each computed usng stereo, as descrbed n [Rander et al., 1996]. Another nput s the optcal flow shown n Fgure 3, whch was computed usng a herarchcal verson of the Lucas-Kanade optcal flow algorthm. The fnal nput s the temporal rate of change of the depth map, estmated from the dfference between two ndependently computed volumetrc models. Fgure 4 shows the result of computng scene flow for the vsble set of ponts on the model. The orgnal ponts are shown n lght grey. The ponts that the orgnal ponts have flowed to by addng the scene flow are shown n black. The darker ponts therefore represent a predcton

5 Orgnal scene ponts Flowed locaton of ponts Fgure 4: The computed scene flow. The orgnal ponts on the model are shown n grey. The locatons that these ponts have flowed to are shown n black. These flowed ponts form a predcton of the model at tme t =2. of the model at tme t = 2. They could be used to enhance the effcency and robustness of shape recovery. In Fgure 4, t s seen that the bendng moton of the player on the rght (the player wth the ball) s recovered, as s the downward (and somewhat sdeways) moton of the ball. The major moton of the player of the left (the player facng the ball) s the upward moton of hs left arm, whch s partally recovered. No flow s recovered for some ponts on the arm because the arm moves very fast relatve to ts sze. Many pxels see completely dfferent surfaces even durng one tme-step. Therefore the rate of change of depth nformaton for the ponts on those surfaces s nvald yeldng no flow estmate for those ponts. 4.2 Known Image Correspondences The second major case s when the structure of the scene s not completely known, but correspondences between mages are avalable. In our calbrated settng, correspondences can be used to compute depth maps, but these depth maps may be too nosy to estmate surface normals and temporal rates of change. Ths stuaton s common. For example, t s typcal n mage based modelng and renderng problems. Whle these problems typcally only consder statc scenes, scene flow can be used as a means of extendng mage based modelng methodologes nto the temporal doman for dynamc scenes. If the surface s not completely known, t s not possble to solve for drectly from one camera. Instead, consder the mplct lnear relaton between the scene and the optcal flow, as descrbed by the Jacoban n Equaton (12). Ths set of equatons provdes two lnear constrants on. Therefore, f we have N > 2 cameras, we can solve, by settng up the system of equatons Bx = U, for Fgure 5: The magntude of the scene flow s dsplayed for ponts on the model (the locatons of whch are obtaned by projectng depth maps from 4 cameras nto the scene). The magntude of the scene flow s dsplayed as ntensty. It can be seen that the largest moton occurs on the ball and the arm of the person at the rear, whle the smallest moton s near the feet of the players. where: B = : : : : : N N N ; U = : N (15) Ths gves us 2N equatons n 3 unknowns, and so for N 2 we have an overconstraned system and can fnd an estmate of the scene flow. (Ths system of equatons s degenerate f and only f the pont x and the N camera centers are co-lnear.) A sngular value decomposton of B gves the soluton that mnmzes the sum of least squares of the error obtaned by re-projectng the scene flow onto each of the optcal flows. We mplemented the above algorthm and appled t to the same sequence that was used n the prevous secton, but wthout usng the surface normal or rate of change of the depth map. We used optcal flows from 15 dfferent cameras. The use of ths many optcal flows ensures that every pont at whch we desre to compute scene flow s vewed by at least 2 or 3 cameras. Fgure 5 shows the magntude of the computed scene flow. The absolute values of the computed flows n the x, y, and z drectons are averaged and dsplayed as the ntensty for each pont. Snce mage correspondences along wth camera calbraton gve us depth maps, the scene flow s computed for a set of ponts, the locatons of whch are obtaned by projectng depth maps from 4 wdely separated cameras nto the scene. It s seen that the moton of the ball,

6 Orgnal Scene ponts Flowed locaton of ponts Flowed locaton of ponts (predcted model) Independently computed model Fgure 6: The ntal pont cloud s shown n lght grey, whle the set of ponts that these ponts have flowed to are shown n black. As can be seen, there s sgnfcant three-dmensonal moton of the player downwards and to ther rght. and the vertcal moton of the left arm of the person at the rear are the most sgnfcant. A close up of the player holdng the ball s dsplayed n Fgures 6 and 7. In Fgure 6, the lght grey ponts represent the model at t =1, whch are dsplaced by the estmated scene flow to gve the darker colored ponts. The same flowed ponts are shown n Fgure 7, except that the lght grey ponts now represent the model at t =2, computed ndependently usng stereo and volumetrc mergng [Rander et al., 1996]. The fgures clearly show that the dsplacement of ponts usng the scene flow results n them movng almost exactly onto the true model. Hence, scene flow may be used as a predctor for the structure of the scene at subsequent tme ntervals. 4.3 No Knowledge of the Surface If the pont x les on the surface, Equaton (12) must hold for every camera. Therefore, t s possble to use the degree to whch Equaton (12) s consstent across cameras as nformaton for a flow-based reconstructon algorthm. Such an approach would, however, be very susceptble to outlers. A sngle large magntude flow whch was wrong could always make the equatons nconsstent. We therefore take a slghtly dfferent approach. The soluton of Equaton (12) can be wrtten n the followng form: du + r (u ) (16)? s the pseudo-nverse r (u ) s the drecton of a ray through the pxel u (see Appen A), and s an unknown constant that depends upon nstantaneous propertes of the surface f. (Equaton (16) holds because Fgure 7: A comparson of the flowed ponts (black) wth an ndependently computed model at the next next tme nstant (lght grey). It s seen that the flowed ponts are a good approxmaton of the model at the next tme nstant. r (u ) s n the null-space ). Therefore, we have the followng constrant on the the scene flow:? du r (u ) m du =0 where m (x) = r (u ) s a vector whch s perpendcular to the plane defned by the camera center and the optcal flow n the mage plane. Hence, f x s actually a pont on the surface, the vectors m (x) should all le n a plane (the one perpendcular to the scene flow ). We form a measure of how coplanar the vectors are by computng the 3 3 matrx: M (x) = X ^m ^m T (18) where ^m s m normalzed to a unt vector. The normalzaton makes the algorthm less susceptble to ncorrect large magntude flows. The smallest egenvalue = (x) of M s a measure of non-coplanarty. We therefore use N, (x) as a measure of the lkelhood that x les on the surface. (N s the number of cameras.) We dscretze the scene nto a three-dmensonal array of voxels, as was done n the Voxel Colorng algorthm of [Setz and Dyer, 1997]. We then compute N, (x) for each voxel, gnorng vsblty as n [Collns,1996]. Ignorng vsblty n ths way does not sgnfcantly affect the performance because our coplanarty measure s not sgnfcantly affected by outlers. (Ths algorthm could be extended to keep track of the vsblty n a smlar manner to [Setz and Dyer, 1997] f so desred.) We present the results of ths algorthm n Fgure 8. We used the data from all 51 cameras of the CMU Vrtualzed Realty dome [Narayanan et al., 1998] (Some of the data

7 that t mght be possble to estmate the scene flow drectly from three such constrants. Unfortunately, dfferentatng Equaton (7) wth respect to x we see that: Fgure 8: A volume renderng of the coplanarty measure N, (x). As can be seen, the gross scene structure s recovered farly well. Note, however, that ths algorthm only recovers structure where the scene s movng. Hence, certan parts of the scene, such as the legs, are not recovered as well as others. The nformaton provded by N,(x) could be combned wth tradtonal sources of nformaton to further enhance the robustness of stereo. from one camera s presented n Fgure 2). For all 51 cameras, we computed the optcal flow from t = 1to t =2. The measure of coplanarty N, (x) was then computed for each voxel and thresholded. Fgure 8 contans a volume renderng of the results. As can be seen, the gross structure of the scene s recovered. Note, however, that ths flowbased reconstructon algorthm can only recover structure where the scene s actually movng. Ths s the reason that certan parts of the scene, such as the legs of the people, are not fully recovered. 4.4 Three-Dmensonal Normal Flow Constrant Optcal flow du s a two dmensonal vector feld, and so s often dvded nto two components, the normal flow and the tangent flow. The normal flow s the component n the drecton of the mage gradent ri, and the tangent flow s the component perpendcular to the normal flow. The magntude of the normal flow can be estmated drectly from Equaton (11) as: 1 jri ri du j =, jri j : (19) Estmatng the tangent flow s an ll-posed problem. Hence, some form of local smoothness s requred to estmate the complete optcal flow [Barron et al., 1994]. Snce the estmaton of the tangent flow s the major dffculty n most algorthms, t s natural to ask whether the normal flows from several cameras can be used to estmate the 3D scene flow wthout havng to use some form of regularzaton. The Normal Flow Constrant Equaton (11) can be rewrtten ri = 0: (20) Ths s a scalar lnear constrant on the components of the scene flow. Therefore, at frst glance t seems ri =,C r ((x; t)[n(x; t) s(x; t)]) : (21) Snce ths expresson s ndependent of the camera, and nstead only depends on propertes of the scene (the surface albedo, the scene structure n, and the llumnaton s), the coeffcents of n Equaton (20) should deally always be the same. Hence, any number of copes of Equaton (20) wll be lnearly dependent. In fact, f the equatons turn out not to be lnearly dependent, ths fact can be used to deduce that x s not a pont on the surface. (See Secton 4.3.) Ths result means that t s mpossble to compute 3D scene flow ndependently for each pont on the object, wthout some form of regularzaton of the problem. We wsh to emphasze, however, that ths result does not mean that s t not possble to estmate other useful quanttes drectly from the normal flow, as for example s done n [Negahdarpour and Horn, 1987] and other drect methods. 5 Conclusons Three-dmensonal scene flow s a fundamental property of dynamc scenes. It can be used as a predcton mechansm to buld more robust stereo algorthms, and for varous scene nterpretaton and renderng tasks. We have presented a framework for computng scene flow from optcal flow, assumng varous nstantaneous propertes of the scene are known. We ntend to extend our framework to ncorporate knowledge of structure computed ndependently at the next tme nstant. We also plan to nvestgate other algorthms for computng scene flow that do not use optcal flow, and develop methods of quanttatvely evaluatng ther accuracy. A Computng u The term s the 3D moton of the pont n the u scene maged by the pxel u. Suppose the depth of the surface measured from the th camera s d = d (u ). Then, the pont x can be wrtten as a functon of P, u,andd as follows. The 3 4 camera matrx P can be wrtten as: P = [R T ] (22) where R s a 3 3 matrx and T s a 3 1 vector. The center of projecton of the camera s,r,1 T, the drecton of the ray through the pxel u s r (u ) = R,1 (u ;v ; 1) T, and the drecton of the camera z-axs s r (0) =R,1 (0; 0; 1) T. Usng smple geometry, (see Fgure 9) we therefore have: x =,R,1 kr (0)k r (u ) T + d : (23) r (0) r (u )

8 Surface f (x,y,z) d Camera P u v r (u ) (0,0) u (u,v ) r (0) Center of Projecton Fgure 9: Gven the camera matrx P and the dstance d to the surface, the drecton of the ray through the pxel u and the drecton of the z-axs of the camera can be used to derve an expresson for the pont x. Ths expresson can be symbolcally dfferentated to gve as a functon of x, : (Care must be taken to choose the sgn of P correctly so that the vector r (u ) ponts out nto the scene.) If camera P s fxed, we have: kr (0)k r (u ) u = r (0) r (u : (24) So, the magntude of s proportonal to the rate of u change of the depth map and the drecton s along the ray jonng x and the center of projecton of the camera. References [Adelson and Bergen, 1991] E. Adelson and J. Bergen. The plenoptc functon and the elements of early vson. In Landy and Movshon, edtors, Computatonal Models of Vsual Processng. MIT Press, [Avdan and Shashua, 1998] S. Avdan and A. Shashua. Non-rgd parallax for 3D lnear moton. In Proc. of CVPR 99, volume 2, pages 62 66, [Barron et al.,1994] J.L. Barron, D.J. Fleet, and S.S. Beauchemn. Performance of optcal flow technques. IJCV, 12(1):43 77, [Collns,1996] R.T. Collns. A space-sweep approach to true mult-mage matchng. In Proc. of CVPR 96, pages , [Costera and Kanade, 1998] J.P. Costera and T. Kanade. A multbody factorzaton method for ndependently movng objects. IJCV, 29(3): , [Horn, 1986] B.K.P. Horn. Robot Vson. McGraw Hll, [Lao et al.,1997] W.-H. Lao, S.J. Aggrawal, and J.K. Aggrawal. The reconstructon of dynamc 3D structure of bologcal objects usng stereo mcroscope mages. Machne Vson and Applcatons, 9: , [Malassots and Strntzs, 1997] S. Malassots and M.G. Strntzs. Model-based jont moton and structure estmaton from stereo mages. CVIU, 65(1):79 94, [Metaxas and Terzopoulos, 1993] D. Metaxas and D. Terzopoulos. Shape and nonrgd moton estmaton through physcs-based synthess. IEEE PAMI, 15(6): , [Narayanan et al., 1998] P.J Narayanan, P.W. Rander, and T. Kanade. Constructng vrtual worlds usng dense stereo. In Proc. of the Sxth ICCV, pages 3 10, [Negahdarpour and Horn, 1987] S. Negahdarpour and B.K.P. Horn. Drect passve navgaton. PAMI, 9(1): , [Penna, 1994] M.A. Penna. The ncremental approxmaton of nonrgd moton. CVGIP, 60(2): , [Pentland and Horowtz, 1991] A.P. Pentland and B. Horowtz. Recovery of nonrgd moton and structure. IEEE PAMI, 13(7): , [Rander et al., 1996] P.W. Rander, P.J Narayanan, and T. Kanade. Recovery of dynamc scene structure from multple mage sequences. In Proc. of the 1996 Intl. Conf. on Multsensor Fuson and Integraton for Intellgent Systems, pages , [Setz and Dyer, 1997] S.M. Setz and C.M. Dyer. Photorealstc scene reconstrcuton by space colorng. In Proc. of CVPR 97, pages , [Sh et al., 1994] Y.Q. Sh, C.Q. Shu, and J.N. Pan. Unfed optcal flow feld approach to moton analyss from a sequence of stereo mages. Pattern Recognton, 27(12): , [Ullman, 1979] S. Ullman. The Interpretaton of Vsual Moton. MIT Press, [Ullman, 1984] S. Ullman. Maxmzng the rgdty: The ncremental recovery of 3-D shape and nonrgd moton. Percepton, 13: , [Waxman and Duncan, 1986] Allen M. Waxman and James H. Duncan. Bnocular mage flows: Steps toward stereo-moton fuson. IEEE PAMI, 8(6): , [Young and Chellappa, 1999] G.S. Young and R. Chellappa. 3-D moton estmaton usng a sequence of nosy stereo mages: Models, estmaton, and unqueness. IEEE PAMI, 12(8): , [Zhang and Faugeras, 1992a] Z. Zhang and O. Faugeras. 3D Dynamc Scene Analyss. Sprnger-Verlag, [Zhang and Faugeras, 1992b] Z. Zhang and O. Faugeras. Estmaton of dsplacements from two 3-D frames obtaned from stereo. IEEE PAMI, 14(12): , 1992.