Scalable Arbitrary Surrounded Surface Calibration for Multi-projector Rendering Application

Transcription

1 Scalable Arbitrary Surrounded Surface Calibration for Multi-projector Rendering Application Qingshu Yuan, Dongming Lu, and Yueming He College of Computer Science and Technology, Zhejiang University, Hangzhou, Zhejiang, P.R.China {ldm, Abstract. A scalable multi-projector calibration method for arbitrary surrounded projection surface is proposed. The method is based on the viewing parameter based principle, which ensures that at each view point and in each direction, the image observed by the camera is the image rendered in these viewing parameters. The principle also ensures that multi-projector can be well aligned. Coded structured light method is used to establish the mapping relation of the projector image coordinates and the camera image coordinates. By prewarp the projection source image according to the mapping relation, image distortions can be removed. A high accuracy mechanical device is used to put the camera on. By moving or rotating the device, camera s extrinsic parameters are easy to be resolved. The method has a build-in support for scalability and the results show that it achieves two-pixel accuracy. Keywords: Multi-projector Calibration, Multi-projector System, Structured Light, Scalable Calibration 1 Introduction High resolution immersive displays involve large field of view using multi-projector mosaics. They are widely used in the areas of heritage presentation, museum exhibition, military training, gaming, etc. Due to the geometric distortions and the geometric overlapping, calibration tasks should first be accomplished, so that the display appears as one single high resolution projector. Calibrations can be achieved by mechanical and electronic alignment [1][2][3]. These approaches align projectors using mechanical devices, and can only be adopted in linear geometric distortions, in other words, only suit for planar surfaces. They require a great deal of personnel work and have assumptions on hardware. Recently camera based calibration techniques become more and more popular. Casually aligned projectors can be calibrated using camera feedbacks. Moreover, the color and brightness imbalance can also be rectified in the same process. It is more convenient and easy to setup and maintain since camera based techniques have no complex mechanical devices. Besides, camera based methods have less costs since costly precise projector mounting devices and tasks are not necessary [4]. The main idea of camera based calibration is to obtain the mapping relation between the 342

2 destination image and the projection image, and then project the pre-warped projection image. To resolve this mapping relation, many methods was brought out. For planar [5] and quadric [6] surfaces, the mapping relation can be described as an equation, and the indeterminate coefficient can be resolved by computing the sampling points. But irregular surfaces cannot be parameterized using a simple equation, so these methods do not fit irregular surfaces. For irregular surfaces, R. Raskar [7] brought forward a method only using one camera. The camera captures some sampled points, and then the mapping relations in these points are computed using coded projector pattern. Mapping relations in rest points are computed by bilinear interpolation. Having the assumption that the irregular surface can be observed in one image frame, this approach cannot be adopted in large scale surface calibration. To say the least, though the large scale surface can be observed in one image frame using wide field of view camera, the calibration precision will be greatly reduced. Moreover, wide field of view cameras have large distortions which will also affect the calibration precision. For scalability in planar surface calibration, Y. Chen et al. [8] proposed a method that use a pan-tilt unit (PTU) to mount the camera on and the PTU allows the camera to move. By moving the camera, the whole surface can be observed. By measuring the error between the two corresponding patter, the system correct the alignment error step by step in one loop. H. Chen et al. [9] presented a method that uses multiple cameras among which one is a root camera, other cameras homography relations are computed related to this one. So the whole surface s mapping relation can be indirectly computed related to the root camera. For arbitrary display surface, there are no appropriate methods to deal with the calibration scalability. We present a scalable arbitrary surface calibration method for multi-projector rendering application. The method has no assumption on the surfaces geometry shape and has the build-in support for scalability in projector numbers. It accomplishes the geometry distortion by pre-warp the projection image and mapping texture by the similarity between the camera image and the projector image according to the viewing parameters of the camera. The rest of the paper is organized as follows. Section 2 will give an overview of basic principle. Section 3 will present the inner projector geometry distortion correction method, followed by the multi-projector surface calibration details, which is presented in Section 4. Section 5 will give an example implementation in our curved surface. And at last we will make some conclusions also with some discussions about the future work in Section 6. 2 The Principle of Viewing Parameters Based Calibration In this section, we will introduce the method s principle of viewing parameters based calibration. For irregular surfaces, surface shape can not be described by an equation. The calibration could be achieved, however, by satisfying the condition that projected images seem to be corrected. That is to say, when someone stands at the best view point position, he or she observes the corrected scene. 343

3 (a) Regular Surface System Fig. 1. Regular and Irregular Surface System (b) Irregular Surface System Supposing that O is the best view point of a multi-projector system, we set O as the origin. Without loss of generality we make coordinate axes X, Y and Z satisfying that XOZ is horizontal and Y s direction is vertically up, which will greatly simplify the implementation. We name these coordinates Physical Coordinates. Rendering applications also have coordinates, which we name as Graphics Coordinates. Images projected are rendered from a fixed view point and in a fixed view frustum frame by frame. In the rendering applications, we will get an image I when we render it from view point O w with the frustum O w A w B w C w D w. Then our viewing parameters based calibration principle can be described as follows: Supposing that the human eye is at O, and watches from the direction corresponding to the frustum OABCD which is congruent with O w A w B w C w D w, and all these points have same coordinate value respectively, he or she observes the scene as if image I is at position ABCD. Since we cannot do a proper quantitative analysis of human eyes, we use a high precision camera to measure instead as Fig. 2 illustrated. To reach the aim previously described, we map the image I pixel by pixel to the projection surface area A B C D show in Fig. 2 (b). So the calibration task can be carried out by satisfying that from any direction and any frustum, each pixel P for the image I is exactly projected at the point P which is the cross point of the line OP with the surrounded projection surface S. For multi-projector systems, since the whole surface is coved by more than one projector, P maybe belongs to more than one projector. Fig. 3 shows the calibration task s details. Y w Objects in Scence Y A! B! A w D w B w I C w A D D! B C C! O w X w O X Z w Z (a) Graphics Coordinates (b) Physical Coordinates 344

4 Fig. 2. Viewing Parameters Based Calibration Principle. Surrounded Projection Surface A! B! E! F! P! D! A D B P C Y C! E H H! F G G! Z O X Fig. 3. Pixel Level Remap in Calibration 3 Inner Projector Distortion Correction using Pre-warping Images Till now, we ve known the aim that we should render some images from graphics coordinates origin and in their frustum and then project them, satisfying that when someone watches in the physical coordinates origin, he or she observes the same image in the corresponding frustum. Images projected onto regular or irregular surfaces, however, will be distorted as shown in Fig. 4 (a). So distortion correction should be rectified by pre-warping the source projection image. For a point P in source projection image and the corresponding point in camera captured image is P, we can establish the mapping relation from P to P, named T. T: (P -> P) (1) However the projection surface may be irregular, therefore T cannot be given by a simply equation. Instead, we use a binary coded structured light method [10] to establish the relation table. Then if we cut a rectified area in captured image, for each pixel Q in it, we can resolve the source projection image coordinates Q in source projection image. Given an undistorted image, for each pixel we calculate the pre-warped pixel in source projection image using T, thus the distortion problem can be solved as shown in Fig. 4 (b). So our aim is changed to be that given an area and any of the inner pixel Q in camera captured image, we find a frustum that contains the area and Q, and then render the image using this frustum. After that, we pre-warp the image using T to find Q. At last we project the pre-warped image source projection image. 345

5 Projection Surface Pre-warp Projection P Source Projection Image Capture P! Camera Captured Image Rectified Q Image Q! Source Projection Image Unused Area Camera Captured Image (a) Distorted Projection (b) Image Pre-warping Fig. 4. Viewing Parameters Based Calibration Principle In implementation, the pixel level correction is resource consuming, so we cut the rectified image into pieces, and map the sub-texture to achieve the pre-warping. For each piece, viewing frustum should be calculated respectively. Fig. 4 shows that two balls are rendered respectively using sub-frustum 1 and 2, and then we remap the image the surface using the method described in Fig. 3. Objects in Scene Objects Shown Projection Surface Subfrustum 1 O Subfrustum 2 (a) Graphics Coordinates Rendering Subfrustum 1 Subfrustum 2 (b) Physical Coordinates Watching Fig. 5. Sub-area Texture Mapping for Pre-warping 4 Multi-projector Surface Calibration Using High Accuracy Mechanical Devices To obtain the surface sub-area s corresponding sub-texture we should first get the sub-frustum of the scene. As described in section 2, a high precision digital camera is used to observe the projection surface, and for each pose, we calculate the frustum using camera s parameters. After rendering an image using the frustum, we obtain the pre-warped image using key point remapping as long as the sub-area built up by the key point is small enough. 346

6 But for large-scale projection surface, the surface cannot be observed in one camera image. If we manually move the camera to another view direction, we will lose the viewing parameters information, as a result, the projection images cannot seamlessly be put together. To get the each sub-surface s view parameters we should bring in a high accuracy mechanical device to move the camera. The inner projector distortions can be corrected using the pixel or texture mapping and the inter projector mosaics are achieved by using the camera s intrinsic and extrinsic parameters. The intrinsic parameters can be obtained by offline camera calibration, while the extrinsic ones can only be obtained by the high accuracy mechanical devices online. The viewing parameters based calibration principle relies on the assumption that the camera s optical center is at the same position. As the camera s size is far smaller than the distance between the surface and the camera, the optical center movement introduced by rotation can be omitted. For each camera pose, we calculate the camera s viewing parameters using the camera s intrinsic and extrinsic parameters, and then render an image using these parameters in the virtual scene. After that, we cut the image into sub-textures and prewarp the image using the method describe in section 3. Then the projectors project the pre-warped image by set the unused area (shown in Fig. 4) to dark. 5 Example Implementation in Curved Surface 5.1 Device Overview The system is composed of three pass six projectors, each pass has two projectors. Here we just talk about the multi-projector calibration, so the stereo vision issues will not be mentioned, that is to say, we just care about 3 projectors. A high accuracy turn plate is used to put the camera on, as shown in Fig. 6. (a) System Hardware (b) Capturing on Turn Plate 347

7 Fig. 6. Device Overview Our method is based on the assumption that any area of the project surface is covered by as least one projector and at most two projectors, as is easy to be achieved by manual placement. The camera should put in the center of the turn plate to satisfy that the optical center is almost at the same position. For simplification of the implementation, the camera and the turn plate should be placed horizontally, which can determined by a gradienter. 5.2 Results As the structured light method can map all the pixels between the camera and the projectors, the overlapped area and inner pixels can also be resolved by the method. But the blending technique is not the key issue in this paper and we will not give the detail of overlap color blending here. We integrate the calibration function to our MultiPro [11] software platform and developed some digital cultural heritage presentation applications as shown in Fig. 7. The result shows that the method can achieve two-pixel accuracy and the calibration can be accomplished in about half an hour which is acceptable for arbitrary surface calibration. 6 Conclusions and Future Works The paper brought forward a viewing parameters based arbitrary surrounded surface calibration method. The method has no assumption on the surface s geometry shape, and has build-in support in scalability. The disadvantage is that the method depends on high accuracy mechanical devices. But for the most cases, a high accuracy turn plate is enough. Moreover, if the camera s extrinsic parameters can be obtained, the mechanical device is not a must, so extrinsic parameters auto calibration is the most important future work. Besides, we will focus on the dynamic view point calibration because in current version, the viewer s view point is fixed at the origin of the Physical Coordinates. (a) Mogao Cave No. 428 of Dunhuang (b) Ancient Hemudu Site Rebuilding 348

8 Fig. 7. Result Applications Acknowledgements The research was supported by the National HI-TECH Research and Development Program of China (2006AA01Z305), the Program for New Century Excellent Talents in University (NCET ) and R&D Program of Zhejiang Province (2005C13024). References 1. Hereld, M., Judson, I.R., Stevens, R.L. Introduction to building projection-based tiled display systems [J]. IEEE Computer Graphics and Applications, 2000, 20(4): 22~ G. Humphreys, M. Eldridge, B. Ian, G. Stoll, M. Everett, and P.Hanrahan. WireGL: A Scalable graphics System for Clusters [A]. In: Proceedings of SIGGRAPH 2001, Los Angeles, USA, ~ K. Li, H. Chen, Y. Chen, D.W. Clark, P. Cook, S. Damianakis, G. Essl, A. Finkelstein, T. Funkhouser, A. Klein, Z. Liu, E. Praun, R.Samanta, B. Shedd, J.P. Singh, G. Tzanetakis, and J. Zheng. Early Experiences and Challenges in Building and Using a Scalable Display Wall System [J]. IEEE Computer Graphics and Applications, 2000, 20(4):671~ Michael Brown, Aditi Majumder, Ruigang Yang. Camera-Based Calibration Techniques for Seamless Multiprojector Displays [J]. IEEE Transactions on Visualization and Computer Graphics, 2005, 11(2) :193~ R. Sukthankar, R. Stockton, and M. Mullin. Smarter Presentations: Exploiting homography in Camera-Projector Systems [A]. In: Proceedings of 8th IEEE International Conference on Computer Vision. Vancouver, Canada, ~ Raskar R, van Baar J, Willwacher T, et al. Quadric transfer for immersive curved screen displays[a]. In: Proceedings of 25th Annual Conference of the Eurographics-Association. Grenoble, France, ~ R. Raskar, G. Welch, and H. Fuchs. Seamless Projection Overlaps Using Image Warping and Intensity Blending [A]. In: Proceedings of 4th International Conference on Virtual Systems and Multimedia. Gifu, Japan, ~ Yuqun Chen, Clark, D.W., Finkelstein, A., Housel, T.C., Kai Li. Automatic alignment of high-resolution multi-projector displays using an uncalibrated camera [A]. In: Proceedings of IEEE Visualization, Salt Lake City, USA, ~ Han Chen, Sukthankar, R., Wallace, G., Kai Li. Scalable alignment of large-format multiprojector displays using camera homography trees[a]. In: Proceedings of IEEE Visualization, Boston, USA, ~ Klaus Strobl, Wolfgang Sepp, Stefan Fuchs. [OL] Camera Calibration Toolbox for Matlab, Yuan QS, Lu DM, Chen WD, et al. MultiPro: A platform for PC cluster based active stereo display system [A]. International Conference on Computational Science and Its Applications. Singapore, PT 1, MAY 09-12, ~