CONCENTRIC MOSAIC COMPRESSION WITH REBINNING OF SLITS (ROSS)

Transcription

1 CONCENTRIC MOSAIC COMPRESSION WITH REBINNING OF SLITS (ROSS) Yunnan Wu a and Jin Li b a Department of Electrical Engineering, Princeton University, Princeton, NJ b Microsoft Research, Bld. 113, Redmond, WA ABSTRACT The concentric mosaic offers a solution for quick construction and easy navigation of a virtual environment. However, the huge amount of data presents a heavy burden. In this paper, we propose the rebinning of slits (ROSS) for the compression of the concentric mosaic. The proposed scheme decorrelates the 3D concentric mosaic by rebinning the vertical slits of the 3D image array into a large 2D panorama, and then compresses the panorama with a JPEG 2000 image coder. Unlike typical video coders that use motion compensation to decorrelate multiple frames, the rebinning converts 3D spatial correlation into stronger 2D correlation. Experimental results show that the ROSS outperforms MPEG-2 coding of the concentric mosaic by an average of 1.0dB. 1. INTRODUCTION The concentric mosaic is very useful for generating photo realistic views of the synthetic and real world. Shown in Figure 1, by rotating a single off-center camera on a round swinging beam and recording the captured images at regular intervals, a concentric mosaic scene is built up quickly. Let the camera shots taken during the swinging of the beam be denoted as C n, where n indexes the camera shot. Let c(n,h,v) be an individual pixel in the camera shot C n, and h and v index the horizontal and vertical position, respectively 1. Alternatively, the concentric mosaic can be interpreted as a series of panoramas P h ={c(n,h,v) n,v}, each of which consists of vertical line of pixels (slits) at position h of all camera shots. The panorama P h can be considered as taken by a virtual slit camera rotating along a circle co-centered with the original beam with an equivalent radius u, the dataset is thus named the concentric mosaic [1]. 1 We use bold capital symbol, such as C n, to represent an image frame. The normal lowercase symbol, such as c(n,w,h), represents an individual pixel. 1

2 Let the horizontal field of view of the camera be FOV, the concentric mosaic can render arbitrary virtual views as long as the viewpoint is within an inner circle of radius r=rsin(fov/2). Rendering the concentric mosaic involves reassembling slits from the captured dataset. Shown in Figure 1, let P be a novel viewpoint and AB be the field of view to be rendered. We split the view into multiple vertical slits, and render each slit independently. For a single slit PV, since the content of the ray does not change unless it is blocked, we simply search for the slit P V in the captured dataset, where P is the intersection point of the ray PV with the camera track. Slit P V approximates what is seen at PV. More detailed description of the concentric mosaic rendering may be found in [1]. Concentric mosaic easily models the 3D environment and enables virtual walkthrough. However, the amount of data is huge. A sample concentric mosaic scene from [1] includes 1350 RGB images with resolution 320x240 and occupies a total of 297MBs. Efficient compression is thus essential. Since the concentric mosaic consists of a sequence of images, existing image/video compression algorithms may be used to compress the dataset. We may compress the concentric mosaic frame by frame, with a still image coder such as the JPEG 2000[3]. Alternatively, we may compress the concentric mosaic with a video coder such as the MPEG-2 [2], which uses frame prediction and residue coding. Usually, by using the cross-frame correlation, the video coder (such as MPEG-2) is much more efficient than the still image coder (such as JPEG 2000) in compressing the concentric mosaic. In this work, we propose an alternative solution. A preprocessing technique is developed to efficiently realign the concentric mosaic dataset from 3D to 2D. We then compress the resultant 2D image with a still image coder. We call the approach rebinning of slits (ROSS). It is shown that with the 2D still image coder, the ROSS is superior to the MPEG-2 in compressing the concentric mosaic. 2. SLITS REBINNING AND COMPRESSION OF THE CONCENTRIC MOSAIC The volume formed by the concentric mosaic dataset is shown in Figure 2. Notice that the top slice in Figure 2 comprises primarily of straight lines with different slopes or slightly bending curves. Such cross-section is called epipolar plane image (EPI). Since each object is captured multiple times during the camera movement, one point on the object leaves a continuous trace in the EPI. As the camera moves, each point of the object 2

3 moves at a different rate depending upon its distance to the camera. The rebinning of slits (ROSS) is developed to stack slits on the EPI trace together. Alternatively speaking, it generates an image panorama that comprises all the content of the original concentric mosaic and yet is as smooth as possible by putting all slits on the EPI trace into the same position of the panorama. After such 3D to 2D conversion, still image compression techniques may be applied to compress the resultant concentric mosaic very efficiently. The work flow of the ROSS algorithm can be illustrated with Figure 3. There are three key operations: rebinning, compression of the rebinned panorama, and compression of the mapping function. We explained the operations in detail below: Step 1. Rebinning. We may perform the rebinning by analyzing the structure of the EPI and rearrange the slits accordingly. However, recovering the structure information is usually difficult. In this work, the rebinning is performed through a simple slit-by-slit insertion. We insert each slit to a best position with regard to certain smoothness criteria. Let F be the rebinned panorama, f(p,v) be an individual pixel in the panorama, where p and v indexes the slit and the vertical position, respectively. We first initialize the rebinned panorama F to the first camera shot, F=C 1. We then go through each camera shot C n ; and within each shot, we go through the individual slit c(n,h,. ) one by one. For each slit, we find an optimal horizontal location p in the rebinned panorama F, and insert the slit c(n,h,. ) between slits p and p+1. Let the insertion location p be recorded through a mapping function G(n,h): p = G(n,h). The optimal insert location p is chosen so that the rebinned panorama F is the smoothest, where the smooth criterion is with a minimum sum of the absolute difference between the inserted slit and its two neighbor slits: min p v { f ( p, v) c( n, h, v) + f ( p + 1, v) c( n, h, v) }. We may exhaustively search the optimal location p in the rebinned panorama. Though such search yields an optimal solution, the computation is intensive and the resultant mapping function G(n,h) is difficult to encode. 3

4 Assuming that the concentric mosaic is shot with a clockwise swinging camera, the panorama will grow towards right. That is, the mapping function of any two slits G(n 1,h 1 ) and G(n 2,h 2 ) satisfy: Constraint 1: Within the same shot, the right slit is mapped to the right. That is: If h 1 >h 2, then G(n,h 1 )> G(n,h 2 ). Constraint 2: For slits at the same horizontal position but at different shots, the slit of the latter shot is mapped to the right. That is: If n 1 >n 2, then G(n 1,h)> G(n 2,h). With the above two constraints, we may restrict the search of the optimal location p of slits c(n,h,. ) start from the right of locations of slit c(n-1,h,. ) and slit c(n,h-1,. ), and greatly speed up the search process: p>max{ G(n-1,h), G(n,h-1) }. (1) Step 2. Coding of the rebinned panorama. After the rebinning process, the JPEG 2000 image coding is used to compress the rebinned 2D panorama. Since the rebinned panorama grows to the right, when we start the rebinning process on camera shot C n, all slits left of G(n,1) in the rebinned panorama F are stable and will not be affected by future rebinning operations. These slits can thus be encoded, which avoids the buffering of a full rebinned panorama. Coupled with the pipeline coding of the JPEG 2000, the ROSS coding of the concentric mosaic can be performed in a relatively small memory footprint. Step 3. Encoding of the mapping function. To recover the concentric mosaic, the mapping function G(n,h) needs to be encoded. For compression efficiency, we choose to encode the inverse of the mapping function {n,h}={g -1 n (p), G -1 h (p)}, where p=g(n,h). With the constraint 1, the left-to-right order of the slits in the same shot is preserved. Therefore, we only need to record the frame shot portion of the inverse mapping function G -1 n (p), as the horizontal index portion G -1 h (p) can be recovered through the number of slits that map to the same frame shot n but is left of slit p: G 1 h ( p) = count p < p ( G ( p ) = G ( p) ) n 1 n 4

5 To further reduce the entropy, we calculate the second order differentials of the shot number H(p)=G -1 n (p) 2G -1 n (p-1)+g -1 n (p-2), and compress the differentials with the LZW algorithm[4]. With the above operation, the mapping function can be encoded at around 2.75 bits per slit, which occupies only a small portion of the overall bitstream. 3. EXPERIMENTAL RESULTS The test datasets are the scene Lobby (1350x320x240) and Kids (975x360x288). We compare the proposed rebinning of slits (ROSS) scheme with two benchmark algorithms. The first benchmark algorithm treats the entire concentric mosaic as a video and compresses it with a MPEG-2 codec. The second benchmark algorithm simply compresses each individual camera shot independently with the JPEG 2000 and concatenated the resultant bitstream. The results are shown in Table 1. It is observed that the ROSS outperforms the JPEG 2000 compression by dB, with an average performance gain of 4.0dB. Since the rebinned panorama in ROSS is also compressed with the same JPEG 2000 codec, the comparison demonstrates the gain achieved through slit rebinning, and shows the effectiveness of the ROSS algorithm. The performance gain of the ROSS versus the MPEG-2 ranges from dB, with an average of gain of 1.0dB. ROSS is thus a very effective way to explore the cross-frame redundancy in the concentric mosaic. 4. ACKNOWLEDGEMENT The authors would like to thank H. Shum, H. Sun and M. Wu for the raw concentric mosaic data. 5. REFERENCES [1] H.-Y. Shum and L.-W. He. Rendering with concentric mosaics, in Computer Graphics Proceedings, Annual Conference series (SIGGRAPH'99), August 1999, pp [2] J. L. Mitchell, W. B. Pennebaker, C. E. Fogg, and D. J. LeGall, MPEG Video Compression Standard, New York, NY: Chapman and Hall, [3] M. W. Marcellin and D. S. Taubman, JPEG 2000: Image Compression Fundamentals, Standards, and Practice, Kluwer Academic Publishers, [4] T. A. Welch, A technique for high performance data compression. IEEE Computer, vol. 17, no. 6, pp. 8-19,

6 Figure 1 The concentric mosaic, capturing & rendering. Figure 2 The data volume of the concentric mosaic. CONCENTRIC MOSAIC JPEG 2000 REBINNING MAPPING FUNCTION G(n,h) LZW BITSTREAM Figure 3 Working flow of the rebinning of slits (ROSS) algorithm. Table 1 PSNR performance (db) for the MPEG-2, JPEG 2000 and ROSS algorithms. LOBBY(0.4bpp) LOBBY(0.2bpp) KIDS(0.4bpp) KIDS(0.6bpp) MPEG JPEG ROSS

7 7