Automatic Projector Display Surface Estimation. Using Every-Day Imagery

Transcription

1 Automatic Projector Dispay Surface Estimation Using Every-Day Imagery Ruigang Yang and Greg Wech Department of Computer Science Univeristy of North Caroina at Chape Hi Abstract Projector-based dispay systems have been used in computer graphics for about as ong as the fied has existed. Whie projector-based systems have many advantages, a significant disadvantage is the need to obtain an accurate anaytica mode of the mechanica setup, incuding the externa parameters of the projectors, and a description of the dispay surface. We introduce a new method for the atter. Instead of empoying some form of imperceptibe structured ight that requires non-trivia infrastructure, we continuay observe images of whatever graphica content is being projected, to refine an ongoing estimate for the dispay surface geometry. In effect we enjoy the benefits of the high signa-to-noise ratio of structured ight, but do not get to choose the structure. he approach is robust and accurate, and can be reaized with commercia off-the-shef components. And athough we do not demonstrate this, it can be extended to incude continua estimation of other system parameters that vary over time. he method can be used with a variety of projector-based dispays, for scientific visuaization, trade shows, entertainment, or the Office of the Future. visuaization, simuation, or entertainment. Exampe systems incude the CAVE [1], the ReActor Room (and simiar systems) by rimensions, the Office of the Future [2], the Princeton Dispay Wa [3], and the Stanford Information Mura [4]. Beyond permanent fixtures, such dispay systems are often used for portabe visuaization, for exampe at conferences or trade shows. On a much more grand scae, newer and more powerfu ight projectors are increasing opportunities to turn arge physica structures into temporary projector dispay surfaces. For exampe, during the miennium ceebration in Egypt, the Pyramids were used as dispay surfaces for dynamic imagery. Whie projector-based systems offer many advantages over other dispay options for many appications, a significant disadvantage is the need to obtain an accurate anaytica mode of the mechanica setup, incuding the externa parameters of the projectors, and a description of the dispay surface. he probem is that the dispay surface is not an integra part of a singe device, and therefore it must be initiay characterized, and periodicay monitored. CR Categories and Subject Descriptors: [Image Processing and Computer Vision]:Digitization and Image Capture Imaging geometry; Scanning; [Image Processing and Computer Vision]: Scene Anaysis Range data; Shape Additiona Keywords and Phrases: Computer Vision, Image Processing, Image-Based Rendering, Shape Recognition Introduction echnoogica and economic improvements are heping to mae projector-based dispay systems increasingy a viabe option for appications such as arge-scae scientific Figure 0. (From eft to right, top to down) (a) A image is directy projected on a curved surface; (b) he image is pre-warped based on the dispay surface estimation. he bending artifact was corrected; (c) A destop window is severey distorted due to a sharp discontinuity on the dispay surface; (d) he view 1/1

2 after correction. We present an iterative approach to automaticay determine the dispay surface geometry, without human intervention, unobtrusivey and continuousy whie the system is being used for rea wor. We use cameras in a cosed-oop fashion to automate the process. Given the physica reationship between projector and a camera, and a crude estimate of the dispay surface geometry, we iterativey refine the estimate based on image-based correation between the nown projector image, and the observed camera image. Specificay we use a Kaman fiter to estimate the ength of a (parametric) ray from each projector pixe. he resut is a compete 3D description of the surface, aowing one to modify the projected imagery so that it appears correct from any given viewpoint [5]. Some experiments resuts are shown in Figure 0. It is interesting to consider the inherent appropriateness of this approach for dispay surfaces. ypicay, finding feature correspondence using correation techniques is ess reiabe for images or regions that ac high-frequency content. However, for our particuar appication, it is OK to miss measurement opportunities in such a region because if there are no probematic features for the system to observe, there are none for the human to observe either. When there are noticeaby distorted features, the user wi see them, but so wi the system, which can then account for them by adjusting the estimate of the dispay surface. Given a sufficient variation of the projected image contents over time, the system eventuay converges on the actua dispay surface geometry. Because our method is non-intrusive, the caibration process can aways been running to maintain an optima caibration whie the system is being used for rea wor. Our simuation resuts (described ater) predict a high degree of accuracy, and our actua impementation appears to agree. Our approach has the foowing ey advantages: Sef-caibrating. Once started, no human intervention is needed. Continuous and unobtrusive. Cose-oop continuous caibration that does not affect the projected image quaity. When there are visibe probems it corrects them, when there are not, it does nothing. Robust. We use a Kaman Fiter (minima mean stochastic estimator) to optimay weight the measured correation, with a reativey conservative tuning to reduce the ieihood of a negative impact from a fase correation. Minima equipment. No need for high-speed cameras or projectors, or speciaized image processing hardware Fexibe setup. he cameras must be rigid but can be ocated reativey casuay with respect to the projectors. he ony restriction is that what they cannot see they cannot be used to caibrate. Stochastic framewor. Because the framewor is in pace, other parameters can be added to the ist of eements to be estimated. For exampe, interna projector parameters coud be estimated using techniques simiar to [6]. Our goa is to improve the setup and maintenance of conventiona projector-based dispay systems, and to further enabe the rendering of perspectivey corrected imagery on more unusua surfaces [2, 5]. Reated Wor We coud categorize different caibration methods based on two orthogona criteria, passive vs. active, and onine vs. off-ine. Active methods usuay emit expicit energy into the dispay environment to aid in the estimation of the surface properties, whie passive methods use ony existing energy in the environment, such as ight. Off-ine caibration methods usuay woud interfere with the norma operation of the system. he norma operation has to be interrupted if an off-ine caibration procedure has to be performed. On-ine caibration methods can be used whie the system is in norma use. Based on these criteria, existing caibration methods can be categorized in tabe 1. On-ine Off-ine Passive Stereo Mechanica aignment Active Imperceptibe structured ight Laser scan, Structured ight abe 1. Different Caibration Method Most commercia systems use a passive off-ine caibration method. hey use precise eectromechanica setup to ensure that projectors and dispay surfaces are compied with the specification, such as perpendicuar projection to a panar surface at a given distance. Such setup is usuay buy and expensive, and sometime impossibe to impement due to space restriction. In some setup, the projectors are casuay paced. In order to generate perspectivey corrected imagery, the exact dispay surface geometry needs to be acquired. Active off-ine method such as Laser range scan or stereo from structured ight can be use. he major disadvantage of these methods is that they interfere with the norma operation. If the system needs to be re-caibrated due to various changes, such as drifts or changes in the setup, it has to be shut down first to perform the caibration. In [2], they proposed a new on-ine active caibration method caed imperceptibe structured ight. hey use specia engineered digita ight projector that is abe to turn ight on and off at a very high rate (over 1000 Hz). his projector projects image bit-pane by bit-pane. wo of the 24 bit-panes are reserved to insert one structured ight pattern and its compement. Because the switching is so fast, human eye is unabe to distinguish between the bit-pane 2/2

3 showing structured ight pattern and the next one that shows its compement. So what a human sees is a norma image with sighty extra shade of gray. However, a synchronized camera with a shutter speed faster than one bit pane s duration is abe to see ony the structured ight pattern. With the hep of these structured patterns, the dispay surface geometry can be accuratey acquired. Because this method hides the patterns within the norma imagery, it can be used onine whie peope are using the system for every day wor. here are two major disadvantage of this method; first, it sacrificed the image quaity, ony 22 of the 24 bit-panes is used to dispay the norma image, the two reserved bit-panes adds an extra gray eve to the entire image, both the coor number and the contrast is reduced; secondy, it requires a specia digita ight projector that is not avaiabe on the maret now. Passive stereo agorithm requires no specia hardware and is very easy to impement. However, finding correspondence between two images based on correation is nown to be unreiabe. So no commercia system to our nowedge has been using stereo agorithm to caibrate their dispay sub-system. he agorithm we proposed in the next section is a passive onine method. Agorithm We define the dispay surface as a trianguar mesh in the projector space. Each vertex (Ver) in the mesh corresponds to a pixe(z) in the projector s image pane. Z is caed a feature point. Feature points are uniformy distributed over the projector s image pane. he density of feature points depends on the surface continuity and the computationa budget. he more irreguar the surface is, the higher density is required, at the cost of onger computing time. he center of projection of the projector (O p ) and Z defines a projection ray, Ver can ony move aong this projection ray. Figure 1 shows the constraint between different points. O p Z Projector Dispay Surface t Ver Z [u 0, v 0 ] Camera Epipoar Line Figure 1. Point Ver is a termination point on the projection ray from O p though Z. It is uniquey decided O c by a parametric vaue t. Ver s projection (Z ) on the image pane is imited by the epipoar constraint. From [7], we now that for a given 3 4 projection matrix (M proj ), and a feature point Z on the image pane, if we rewrite M proj as M [ ~ proj = p p], where p is a 3 3 matrix, p ~ is a 3 1 vector, Ver can be computed by the foowing formua: x y = p z 1 u ~ ( ~ p + t ~ v ) 1 where t is a parametric vaue, and on the projector s image pane.. Z u v~ ] Equation 1 = [ ~,, its position When a feature point Z is projected on the dispay surface, its projection Z can be found by a camera. If we now the projection matrix of the camera (M cam ), we coud sove for t using the traditiona stereo agorithm. hough stereo agorithm is simpe, it ony uses a singe observation, thus it is very vunerabe to noise or fase matches. So instead, we use a Kaman fiter, a minimum variance stochastic estimator, to estimate the parametric vaue t iterativey. A basic introduction to the Kaman fiter can be found in Chapter 1 of [8] and [9]. With mutipe observations over time, the chance of a fase match is greaty reduced so the robustness of the agorithm is improved. he Kaman fiter aso provides a predication of where the Z wi be projected. We use this predication to search for match in a smaer area whie the stereo agorithm usuay has to search the entire epipoar ine. When an actua match is found, the difference between the predication and the actua match was corrected. In our Kaman fiter mode, for every Z, we have a measurement [ u0, v ] 0 -- its projection on the dispay surface observed by the camera, we want to sove for t, the parametric vaue that determines the 3D position of Ver. Assuming the dispay surface is static, the foowings are our system equations that govern the estimate process. Because the perspective projection is not inear, we have to use Extended Kaman fiter (EKF). t = t + 1 [ u, v, s] = M cam [ x y [ U, V ] = [ u0, v0] + H ( ) where H is the Jacobian matrix of partia derivatives of measurement function with respect to t, ( u ) H = s t z 1] ( v ) s t. Equation 2 3/3

4 We denote P as the estimated error covariance, R is the measurement variance, the time update equations are t+ 1 = t Equation 3 P = P + Q + 1 And our measurement update equations are R 0 K = P H ( H P H + ) 0 R t = t + K ([ u v ] u v 0, 0 [, ] ) s s P = ( I K H ) P 1 Equation 4 After the Kaman fiter update, the 3D position of Ver is updated with the new t using equation 1. Our agorithm starts with a very rough estimate of the dispay surface -- every feature point has the same initia t set to 0.5, then randomy refines the estimation iterativey. In each iteration, we perform the foowing operation: 1. Capture of dispay image and the projected imagery 2. Seect a subset of feature points to find their corresponding image in projected image using correation. 3. Kaman Fiter update of these seected feature points. 4. 2D Deaunay rianguation to update the mesh. We et this process run as ong as the system is turned on. Notice that in the time update equation, we added a very sma amount of process variance to compensate for possibe drifts in the system. If we set Q to zero, the Kaman fiter wi not update its estimate of the dispay surface after it has converged. With this added process variance, it is sti abe to adjust itsef, though very sowy, towards changes due to drift or other factors, even after convergence. Seection of Feature Points Due to computationa constraints, we cannot compute the entire feature set within in one iteration, instead we seect a subset of feature points in one iteration. he seection process has two parts, sequentia seection, and distance-based seection. In sequentia seection, we first define a ist of feature points, and then permute the ist. In each iteration, a number of consecutive points in the permuted ist are seected. So that every point has exacty equa possibiity of being updated. In distance-based seection, we want to identity possibe outying points and correct them as soon as possibe. We found that outying points are usuay far away from the correct points in 3D. So we use a seection process based on Eucidean distance. We define a maximum neighborhood distance (MND). For every feature point (Z) that has been updated, we find its cosest neighbor (Z n ) that aso has been updated at east once, if the distance between Z and Z n is greater than MND, this Z is considered as a point with higher uncertainty and added to the seected point ist. One may argue that this distance-based seection imposes an assumption of the dispay surface geometry -- no two neighbor points can be farther than MND, but in fact, this seection ony tries to identify possibe outying points. If a point with high uncertainty turns out to be a correct one indeed, it wi converge to that position in subsequent updates. In practice, we set the MND to be twice the distance between two neighboring feature points with the initia estimate (t = 0.5). We found this MND wors we in practice. Predicative Pattern Match Once we have a ist of seected feature points, we want to find out its corresponding points in the camera image; With the current parametric vaue t and the estimated error variance P, we compute the cosest point ( Z min ) and the furthest point ( Z max ), where Z min is computed with t - sqrt( P ), and Z max is computed with t + sqrt( P ); he Z max and Z min are projected bac to the camera image pane. hey form a rectanguar bounding box on the image pane. One diagona ine of the bounding box is the epipoar ine. he search ony needs to be performed in the bounding box. he estimated error variance P wi graduay decrease as the Kaman fiter sowy converges. Consequentiay, the search area wi become smaer and smaer. For each seected feature point, we use a 16x16 boc around it as the correation tempate. Matrox Imaging Library (MIL) is used to perform the pattern matching within the specified bounding box in the camera image. It can return a match with subpixe accuracy. In some cases, there wi be mutipe matches returned by the MIL, a are within the bounding box. In such a case, we computed the mean and the standard deviation of these matches, if the standard deviation is greater than the measurement variance R, the entire match set is discarded. Otherwise, their mean is used as the fina resut. Since MIL s tempate matching routine searches the entire area, some time it wi return a matching that is not on the epipoar ine. his is probaby due to two factors; first, the caibration error of the projector and camera s externa and interna parameters; secondy, there is eectronic noise during the process of digitizing anaog videos. If such deviated resut is encountered, we compute its distance to the epipoar ine (one diagona ine of bounding box); if it is greater than R, this match is discarded. 4/4

5 great engths to achieve fast rendering speed. We just wrote a bare bone OpenGL program that was enough to demonstrate that the surface estimate was correct. his can be seen in video. A faster computer woud do it. Further more, we beieve that in most appications, the rendering and the capture of depth information is decouped. Experiment Resuts Figure 2. A image captured by the camera. he bounding box is superimposed, its diagona is the epipoar ine. he highighted green square indicates an accepted match, whie the red circe indicates a rejected one because it is too far away from the epipoar ine. In our agorithm, we do not perform the rectification of the camera image. Rectification is widey used in stereo agorithm. It is a two-dimensiona transformation that transfers the epipoar ines parae to the image scan ines. So that the search for correspondence is imited to the scanines. We do not do so because the number of points we compute in each iteration is at most at the order of ten, rectification woud cost more time than the speedup it brings in the search phase. We use MIL to perform tempate matching, its hierarchy search agorithm is very fast so that there is virtuay no time difference whether the search area is a ine or a box. Pus, we sti chec the resut returned by MIL to see if it is within the epipoar ine with some toerance. Rendering Correct Image We use the technique described in [2] to render perspectivey corrected image. o do this, we first need to create a trianguar mesh. We first impemented a scan-ine based trianguation routine to create a compete mesh and et it deform as its vertices 3D position is being updated. In our experiments, we found that this approach created some very noticeabe distortions if there is a hoe in the mesh. he hoe can be defined as one not-yet-updated point surrounded by updated points. So we have to perform trianguation in run-time. Assuming there is no sef-intersection of the dispay surface, the trianguation can be performed in projector s screen space. 2D Deaunay trianguation agorithm is much easier to impement and more robust than its 3D counterpart. Since rendering is not our primary focus, we did not go to We impemented our agorithm under Windows N environment. We initiay deveoped and tested our agorithm in simuation, in which we performed some we-controed experiments, and then went to a rea system. he difference between the simuator and the rea setup is that in the rea setup, we have to find the externa and interna parameters of the camera and the projector. We first present our resut in the simuator, then the resut in a rea setup. o mae our resut more convincing, in our simuation, we used the rea externa and interna parameters of the camera and the projector found in the rea setup. So a of our experiments have the same setup, the projector is about 1000 mm away from the dispay surface, and the camera is 600 mm upper right to the projector, pointing at the dispay surface. In our simuation, we set Q (process noise) to 1e-10, measurement variance R to 9 (3*3 pixe), and the estimated error covariance P to he density of the feature points is 40 x 30. We performed two experiments, one with a panar dispay surface with discontinuity, the other with a curved concave surface. During the estimation process, we happened to use a canned image sequenced recorded using a DV camcorder. In practice, this sequence woud be the ongoing stream of whatever every-day imagery the user was dispaying. We et the system run about 45 minutes in each experiment, the accuracy of our resut is shown in abe 2. Mean Error(mm) Max. Error 1 (mm) Panar Surface Curved Surface abe 2. Accuracy of the Simuation he estimated surfaces are shown in figure 3 and 4 respectivey. Figure 0 (c) and (d) are simuated pictures based on the estimate of the panar surface. 1 In both of our experiments, we found a few outies (five or six), a of them were on the boundary. hey are at east 100 mm away from their neighbors, so they can easiy be identified by the distance-based seection routine described earier. he resut shown here does not tae into account the points seected by distance-based seection routine. 5/5

6 Figure 3. Panar surface simuation. Bue surface is the actua surface; red dots are the estimated feature points. Light bue dots are seected outying points detected by our distance-based heuristic. Figure 5. he estimate of a curved dispay surface after we run our agorithm for over a haf hour in a rea setup. Concusions Beyond arge dispay systems, we are excited by the growing prospect of graphica imagery dispayed on rea surfaces around us [2]. We beieve that our approach to surface estimation provides an important piece of the puzze. he approach is accurate, robust, and can be impemented in practice with reasonaby common components and minima infrastructure. Figure 4. A bird-eye view of the curved surface simuation. In our rea setup, there is no ground truth we can compare our resut with. he estimated surface in the rea setup is shown in figure 5. Figure 0 (a) and (b) show the difference between an uncorrected view and a corrected view. As they show, the distortion due to non-panar dispay surface was corrected. More of the resuts can be found in the video. Beyond the agorithmic improvements we present here, we oo forward to improved hardware. For exampe, some day smart projectors with buit-in cameras wi be common, enabing automatic adjustments beyond simpe eystone correction. Some day graphics engines wi support more efficient rendering onto non-panar (and non-rectanguar) surfaces, and maybe wi even support automatic view-dependent correction. References [1] Curz-Neira, Caroina et a., 1993 Surround-Screen Projection-Based Virtua Reaity: he Design and Impementation of the CAVE, SIGGRAPH Conference Proceedings, Annua Conference Series, Addison-Wesey, Juy [2] Rasar, R. et a., 1998, he Office of the Future: A Unified Approach to Image-Based Modeing and Spatiay Immersive Dispays. SIGGRAPH Conference Proceedings, Annua Conference Series, Addison-Wesey, Juy /6

7 [3] Scaabe Dispay Wa. [4] Greg Humphreys and Pat Hanrahan, 1999, A Distributed Graphics System for Large ied Dispay, in the Proceeding of IEEE Visuaization 1999.Oct [5] R. Rasar, M, Cutts, G. Wech, and W. Stürzinger, 1999, Efficient Image Generation for Mutiprojector and Mutisurface Dispay, in the Proceedings of the Ninth Eurographics Worshop on Rendering, (Vienna, Austria), June 1998 [6] Azarbayejani, Ai, and Aex Pentand, Juen 1995, Recursive Estimation of Motion, Structure, and Foca Length. IEEE rans Pattern Anaysis and Machine Inteigence, June 1995, 17(6). [7] O. Faugeras, hree-dimensiona Computer Vision: A Geometric Viewpoint,Cambridge, Massachusetts; MI Press, [8] Maybec, Peter S Stochastic Modes, Estima-tion, and Contro, Voume 1, Academic Press, Inc. [9] Greg Wech and Gary Bishop, An Introduction to Kaman Fiter. echnica Report, R , Dept. of Computer Science, University of North Caroina at Chape Hi, Chape Hi, NC 7/7