Virtual Exhibition of Traditional Dances by Blending Colors of Multiple Images

Transcription

1 Virtual Exhibition of Traditional Dances by Blending Colors of Multiple Images Takeshi Shakunaga Yasuhiro Mukaigaa Ryo Yamane Daisuke Genda Yuji Kamon Department of Information Technology, Okayama University, JAPAN Abstract A color blending method for generating a high quality image sequence of human motion is presented. The 3D human shape is reconstructed by volume intersection and expressed as a set of voxels. As each voxel is observed as different colors from different cameras, voxel color needs to be assigned appropriately from several colors. We present a color blending method hich calculates voxel color from a linear combination of the colors observed by multiple cameras. The eightings in the linear combination are calculated based on both viepoint and surface normal. As surface normal is taken into account, the images ith clear texture can be generated. Moreover, since viepoint is also taken into account, high quality images free of unnatural arping can be generated. To examine the effectiveness of the algorithm, a traditional dance motion as captured and ne images ere generated from arbitrary viepoints. Compared to conventional methods, quality at the boundaries as confirmed to be improved. Introduction Recently, a lot of research has been conducted into preserving cultural treasures in digital archives[, 2]. In particular, intangible cultural treasures, such as traditional dance, are difficult to preserve because a complete archiving method has not been developed. To capture human motion, a marker-based motion capture system is often used. This system can precisely measure the 3D position of markers, and the measured data can be used in many applications, including computer graphics and motion analysis. Although costumes and ornaments are also part of intangible cultural treasures, this visual information cannot be captured by ordinary motion capture systems. If the 3D visual information of dance motion as archived, the viepoint could be controlled arbitrarily. Control of the viepoint ould be important for both digital archiving and dance training. To generate a ne image hich can be observed Camera PC S-Video Netork Synchronization Signal Figure : Multiple camera system. from an arbitrary viepoint, many methods using multiple cameras have already been proposed[3, 4, 5]. A 3D model of the human body is reconstructed based on images, and the surface texture is mapped onto the 3D model. While each point on the human body is observed from several cameras, the colors are not same. Therefore, an appropriate surface color needs to be calculated from several colors. To reproduce a detailed texture, calculation methods based on the surface normal have been proposed[6]. Hoever, these methods tend to be noisy because the camera from hich the color is taken is different for each voxel. Other calculation methods based on viepoint have been proposed to improve the quality of images generated[7, 8]. These methods, hoever, often cause unnatural arping of texture, especially at boundaries. In this paper, e propose a ne color blending method based on both viepoint and surface normal. As this approach does not produce unnatural arping at boundaries, the quality of generated images is improved. 2 Image generation 2. Multiple camera system In this paper, e assume that a target person dances at the center of a room, and that multiple cameras are installed to capture image sequences from var-

2 ious angles, as shon in Fig.. The image sequences are synchronized, ith all cameras geometrically calibrated in advance. 2.2 Purpose In this paper, e do not treat the 3D reconstruction problem. We assume that the 3D shape is reconstructed by the volume intersection method. The reconstructed shape, knon as the visual hull [9, ], is larger than the true shape. We regard the visual hull as the human body. The 3D shape is expressed using a set of micro cubes called voxels. Since each voxel contains 3D information, it is easy to geometrically calculate the 2D position in a ne image from an arbitrary viepoint. Hoever, it is not easy to decide the color of the voxel, because each voxel is observed from several cameras and the observed colors are usually different. Our aim is to decide an appropriate color for each surface voxel. In this paper, the folloing notation is used: surface voxel: s unit surface normal vector of s: N virtual viepoint: eye camera: C j ( j p) here v j is a variable hich specifies the visibility of the voxel as follos: { : s is visible from Cj. v j = (2) : s is invisible from C j. The f is a function hich takes arguments of surface normal, camera direction, and viepoint. Using the eightings, the color I of the surface voxel s is calculated by I = p j= j I j, (3) here is a normalization factor equal to the sum of the eights. If the sum is zero, the surface cannot observed from any camera, and so this case can be ignored. C 3 eye C 2 θ N N C C 4 C 8 s observed color of s from C j : I j unit vector from C j to s: V j unit vector from eye to s: V To make the problem easier, 2D conditions in hich all cameras, viepoint, and voxels are on a single plane is considered, as shon in Fig. 2. In this system, absolute angles of surface normals and the viepoint are specified by θ N and θ eye, respectively. Of course, this can easily be extended to the 3D case. 2.3 Linear combination of observed colors Although each surface voxel is observed by several cameras, the colors are not the same because of complex factors including specular reflections, occlusions, errors in 3D shape reconstruction, and characteristics of the cameras. Therefore, it is difficult to analyze color differences precisely. In our method, voxel color is calculated by color blending instead of analysis by assigning a eight parameter j to each camera C j, ith the eight parameter given by: j = v j f(n s, V j, V ), () C 5 C 6 C 7 Figure 2: Top vie of the system. 2.4 Conventional methods To decide the value of j, the function f needs to be appropriately designed. Conventional methods can be classified into to categories, surface normal-based and viepoint-based. Details of these algorithms are given in the folloing sections Calculations based on surface normal To acquire detailed textural information about an object, an image should be taken from the surface normal, because the surface appears largest from this direction. Weightings are then calculated by the folloing function based on surface normal[6]. f(n, V j, V ) = { : if N Vj is the smallest. : otherise (4)

3 Clearly, this function does not depend on the viepoint θ eye. Although not blurred, the texture generated using this function tends to be noisy, because the camera from hich the color is taken is different for each voxel Calculations based on viepoint To improve the quality of the generated textures, the camera nearest to the viepoint should be selected. If the angle beteen the camera and the viepoint is small, forced arping of the texture can be avoided. To assign large eightings to the nearest cameras, interpolation methods are often used[, 2]. The function f is thus taken to be the linear interpolation of the colors from the to cameras C l and C r as follos[7]: α : C l f(n, V j, V ) = α : C r (5) : otherise here α ( α ) is a parameter representing the position of the viepoint. As an alternative, Matsuyama and Takai[8] have proposed a calculation method using the m-th poer of the inner product of V j and V as follos: f(n, V j, V ) = (V j V ) m, (6) here m is a parameter that controls the degree of color blending. Figures 3 (a) and (b) sho eightings as a function of θ eye as determined by linear interpolation and by the m-th poer of the inner product, respectively. In the figure, different cameras are shon in different colors, as shon in Fig.2. Clearly, functions based on viepoint are independent of surface normal θ N, ith the same eightings assigned to all voxels even if the surface normals are different. Hence, the generated texture is smooth, and as the number of cameras increases, the texture becomes more realistic. These functions, hoever, often cause unnatural arping of textures, especially at boundaries. If a surface is observed from the perpendicular to the surface normal, the surface is almost invisible, and this texture is forcedly arped. This problem mainly occurs at boundaries. 2.5 Calculations based on both viepoint and surface normal The eak point of surface normal-based methods is that the generated texture tends to be noisy, hereas (a) (b) Figure 3: Weightings as a function of θ eye. (a) Linear interpolation of to cameras. (b) m-th poer of the inner product (m = 5). the eak point of viepoint-based method is that unnatural arping occurs at boundaries. A ne function is thus proposed based on both viepoint and surface normal. In this method, the function f can be expressed as the product of to functions, f eye and f N as follos: f(n, V j, V ) = f eye (V, V j )f N (N, V j ). (7) The function f eye needs to be large hen the angle beteen V and V j is small, hereas the function f N needs to be large hen the angle beteen N and V j is large. Although a linear interpolation or an inner product could be used, it is difficult to control the degree of color blending. Only to colors are used in the linear interpolation, but too many colors are used in the inner product. Therefore, m and n-th poers of inner products are chosen as f eye = (V j V ) m, (8) f N = ( N V j ) n, (9) here only the parameters m and n remain to be decided. Figure 4 shos the behaviors of the function for values of m of, 5 and 2. Values of j are shon in the left set of graphs, and normalized values are shon in the right. If m is small, too many colors are blended and the generated texture ill be blurred, hereas if m is large, a single color is used primarily. Thus m has a large impact on texture, and both m and n are set to 5 hen the angle beteen cameras is 45. If there are more cameras, the m and n should be smaller. Graphs on the right of figures 5, 6, 7 and 8 sho the eightings assigned to each camera for a variety of different methods. Graphs on the left sho the eighting of a single camera, 4. The right color maps indicate the eightings of cameras by color. The relationship

4 .8 Σ (a) m = Σ Figure 5: Weightings based on surface normal (b) m = 5 Σ Figure 6: Weightings based on viepoint (linear interpolation) (c) m = Figure 4: Weightings calculated from the m-th poer of the inner product. beteen color and camera is shon in Fig.2. The proposed method can be seen to take both viepoint and surface normal into account. 3 Experimental results 3. ARAMAI data To examine the effectiveness of the proposed method, a studio including eight cameras (SONY DXC-2A) and eight PCs as constructed to capture image sequences. The observing area is a 2m 2m 2m cube. Image sequences are captured synchronously by the cameras at video rate (3 fps) ith an image size of Since the idth of one pixel corresponds to 5mm at the center of the room, voxel size is set to 5mm 5mm 5mm. To easily separate human region from the background, the floor and alls ere covered by green cloth. As an archive of intangible cultural treasures, a Japanese traditional dance ARAMAI as captured. Because the dance is very fast, hair and costume shapes change drastically. Figure 9 shos an example of input images at t = (here t is the frame number) Figure 7: Weightings based on viepoint (5th poer of the inner product) Figure 8: Weightings based on both surface normal and viepoint (proposed method). 3.2 Comparison of generated images Figures 2 and 3 sho images generated for θ eye = at t = and for θ eye = 5 at t = 244, respectively. The middle ro shos enlargements of the rectangle regions in the upper ro of images. The loer ro shos camera eightings for each surface point by color. The left column shos results generated using

5 Figure 9: Example of input images (t = ). the conventional method based only on surface normal, the center column shos results based only on viepoint (the 5th poer of the inner product), and the right column shos results generated using the proposed method (m = 5, n = 5). The quality of the texture is much improved from the others at the boundaries. 3.3 Virtual exhibition of ARAMAI 3.3. Camera control independent of dancer Figure shos an example of generated movie ith changing virtual viepoint. In this example, the camera position is moved at a constant speed around the dancer. We have confirmed that the position of the virtual camera doesn t affect the quality of the generated images Camera control by motion analysis Figure shos another example of generated movie. In this example, a virtual camera position is controlled as if the camera alays keeps in front of the dancer. In order to implement the camera ork, e have utilized a motion data hich as independently captured by a hand-made motion capture system. That is, the motion data as captured hen the same dancer danced Aramai again here she ore a special dress ith fifteen color markers. After analyzing her posture in each frame, e can easily calculate the front direction in each frame. Thus, e can also estimate a pose of the visual hull in each frame by matching the hull to a motion data in 3-d space. Although the pose estimation is somehat rough, e can synthesize a very useful camera ork from the estimated poses. Figure 2: Comparison of generated images (θ eye =, t = ). In Fig., the first and third ros sho a generated movie, hile the second and fourth ros indicate estimated poses of the upper image. 3.4 Quantitative comparison Finally, a generated image as compared to a real image for quantitative comparison. An additional camera as installed and a real image as captured, as shon in Fig.4(a). The virtual viepoint as then set to be the same as the additional camera. Images captured by the additional camera ere not used for image generation. Figures 4(b), (c) and (d) sho results of the surface normal-based method, the viepoint-based method and the proposed method, respectively. For numerical evaluation, differences beteen the real image and the generated images ere calculated, as shon in (e), (f) and (g). RGB root mean square errors of each method are shon next to the images. Comparison of (e) and (g) reveals that errors are reduced by the proposed method. Further comparison of (f) and (g) reveals that the proposed method can improve the quality of texture at boundaries. 4 Conclusions In this paper, a method for generating high quality images by blending the colors observed from multiple cameras as proposed in hich eightings of the

6 t = t = 2 t = 4 t = 6 t = 8 t = t = 2 t = 4 t = 6 t = 8 Figure : A movie generated by a virtual camera moving around the dancer. To examine the effectiveness of the proposed method, 3D motion and visual information of the Japanese traditional dance ARAMAI as captured using eight cameras. The quality of images generated from arbitrary viepoints is obviously improved over the conventional methods. In this paper, only the color blending problem as discussed ith neither color calibration beteen cameras nor fine 3D reconstruction. Hoever, these problems need to be solved exhaustively in the future. We intend to analyze differences in colors, and improve the quality of output images. Acknoledgments The authors ould like to thank WARABI-ZA ( for their kind cooperation in the capture of the ARAMAI dance. This research has been supported by Japan Science and Technology Corporation under Ikeuchi CREST project. Figure 3: Comparison of generated images (θ eye = 5, t = 244). cameras ere calculated based on both viepoint and surface normal. By changing the exponent parameters m and n, the degree of dependence on viepoint and surface normal can be controlled. References [] D. Miyazaki et al.: The Great Buddha Project: Modeling Cultural Heritage through Observation, Proc. the Sixth International Conference on Virtual Systems and MultiMedia (VSMM 2), pp.38-45, 2. [2] M. Levoy et al.: The Digital Michelangelo Project: 3D Scanning of Large Statues, Proc. SIGGRAPH 2, pp.3-44, 2. [3] T. Kanade, P. Rander and P. J. Narayanan: Virtualized Reality: Constructing Virtual Worlds from

7 t = t = 2 t = 4 t = 6 t = 8 t = t = 2 t = 4 t = 6 t = 8 t = 2 t = 22 t = 24 t = 26 t = 28 Figure : A movie generated by a virtual camera in front of the dancer.

8 [] S. M. Seitz and C. R. Dyer: Vie Morphing, Proc. SIGGRAPH 96, pp.2-3, 996. [2] S. E. Chen and L. Williams: Vie Interpolation for Image Synthesis, Proc. SIGGRAPH 93, pp , 993. (a) real image (b) (c) (d) (e) 5.3 (f) 4.6 (g) 4.2 Figure 4: Evaluation. (a) real image. (b),(c) and (d) reconstructed images. (e),(f) and (g) differences beteen reconstructions and the real images, ith RGB RMS differences also shon. Real Scenes, IEEE MultiMedia, Vol.4, No., pp.34-47, 997. [4] S. Moezzi, L. C. Tai and P. Gerard: Virtual Vie Generation for 3D Digital Video, IEEE MultiMedia, Vol.4, No., pp.8-26, 997. [5] I. Kitahara, H. Saito, S. Akimichi, T. Ono, Y. Ohta and T. Kanade: Large-scale Virtualized Reality, Proc. CVPR2, Technical Sketches, 2. [6] T. Takai and T. Matsuyama: Interactive Vieer for 3D Video, Proc. Fourth International Workshop on Cooperative Distributed Vision, pp , 2. [7] H. Saito, S. Baba, M. Kimura, S. Vedula and T. Kanade: Appearance-Based Virtual Vie Generation of Temporally-Varying Events from Multi-Camera Images in the 3D Room, Proc. Second International Conference on 3-D Digital Imaging and Modeling (3DIM 99), pp , 999. [8] T. Matsuyama and T. Takai: Generation, Visualization, and Editing of 3D Video, Proc. Symposium on 3D Data Processing Visualization and Transmission, pp , 22. [9] W. Matusik, C. Buehler, R. Raskar, S. J. Gortler and L. McMillan: Image-Based Visual Hulls, Proc. SIG- GRAPH 2, pp , 2. [] A. Laurentini: The Visual Hull Concept for Silhouette-Based Image Understanding, IEEE Trans. PAMI, Vol.6, No.2, pp.5-62, 994.