INTRA-FRAME PREDICTION FOR HIGH-PASS FRAMES IN MOTION COMPENSATED WAVELET VIDEO CODING

Transcription

1 INTRA-FRAME PREDICTION FOR HIGH-PASS FRAMES IN MOTION COMPENSATED WAVELET VIDEO CODING LESZEK CIEPLINSKI, JORDI CABALL, SOROUSH GHANBARI Mitsubishi Electric ITE-VIL, 20 Frederick Sanger Road, Surrey Research Park, Guildford, Surrey, GU2 7YD, United Kingdom Abstract: Key words: In the existing inter-frame wavelet video coding techniques, the poorly matched pixels from adjacent frames are temporally filtered into the high-pass temporal frame in the same way as the good matches using motioncompensated prediction (MCP). This tends to result in high-energy blocks being introduced into the high-pass frame, which leads to less efficient spatial coding of the affected frames. We propose to use prediction from spatial neighbouring pixels for high-pass filtering of the poorly matched blocks, which results in a significantly improved performance at low bitrates with no loss at high bitrates. Scalable video coding, wavelets, MCTF, prediction, spatial correlation 1. INTRODUCTION Highly-scalable video coding based on motion-compensated spatiotemporal wavelet decomposition has recently become a very active area of research. The principal characteristics of this approach are its high coding efficiency, which is competitive with the state of the art codecs such as MPEG-4 AVC 1 and the flexibility of adaptation to different network characteristics, varying bandwidth, error conditions, and receiver capabilities. In the existing inter-frame wavelet video coding techniques (e.g. Ohm 2, Hsiang 3 and Taubman 4 ), the poorly matched pixels from

2 EPLINSKI, JORDI CABALL, SOROUSH GHANBARI adjacent frames are temporally filtered into the high-pass temporal frame in the same way as the good matches using motioncompensated prediction (MCP). This tends to result in high-energy blocks being introduced into the high-pass frame, which leads to less efficient spatial coding of the affected frames. We propose to use prediction from spatially neighbouring pixels for high-pass filtering of the poorly matched blocks, which results in a significantly improved performance at low bitrates with no loss at high bitrates performance. 2. INTRA-FRAME PREDICTION The motion-compensated temporal filtering (MCTF) plays an important role in motion compensated 3-D wavelet video coding in terms of coding efficiency and temporal scalability. Since MCTF is a wavelet transform, it can be implemented by using the lifting approach 4. In the following, the Haar temporal wavelet transform is examined and the notation introduced by Ohm 2 is followed. For two successive frames A and B, the high pass filtering equation is given by Eq. (1), where A (current frame) and B (reference frame) are the original frames and H is the high pass-filtered frame, m and n index the pixels and k and l are the motion vector components. The low pass filtering is adapted for connected pixels (Eq. 2) and for covered/uncovered pixels (Eq. 3). H(m, n) = ( 2/2) * [A(m,n) B(m - k, n l)] (1) L(m, n) = H(m + k, n + l) + 2 * B(m, n) (2) L(m, n) = 2 * B(m, n) (3) At the decoder, by using L and H, we can perform the same interpolation on H and reconstruct B for connected pixels exactly (Eq. 4) and for covered/uncovered pixels (Eq. 5). B(m, n) = (1/ 2) * [L(m, n) H(m + k, n + l)] (4) B(m, n) = (1/ 2) * L(m, n) (5)

3 Intra-frame prediction for high-pass frames in motion compensated 3 wavelet video coding After B is available, we can perform the same interpolation on B as was performed at the encoder, and reconstruct A exactly (Eq. 6). A(m, n) = 2 * H(m, n) + B(m - k, n l) (6) For areas with occlusions and non-homogeneous motion fields the spatially neighbouring pixels may constitute better predictors than the temporally neighbouring frame. Therefore, we introduce a set of new prediction modes for covered/uncovered blocks (generally referred to as unconnected blocks) in the high pass frames, in which the spatial neighbourhood is used for prediction. The application of this intra-frame prediction by using pixels from the same frame near to the blocks to be predicted helps to improve the quality (i.e. reduce the energy) of high pass temporal frames. It also reduces the number of motion vectors that need to be transmitted, which leads to better performance at low bitrates. When an intra-prediction mode is used, the motion-compensated temporal filtering (MCTF) is slightly modified in lifting implementation for the corresponding blocks in two successive frames A (current) and B (reference). The high pass filtering H is adapted for unconnected pixels as in Eq. (7), where A(m, n) are the prediction pixels from the current frame. The high pass filtering for connected pixels and the low pass filtering remain the same. The current frame is then reconstructed for unconnected pixels by Eq. (8), where A(m, n) are the previous reconstructed pixels. The remaining reconstruction equations remain unchanged. H(m, n) = ( 2/2) * [A(m,n) A(m, n)] (7) A(m, n) = 2 * H(m, n) + A(m, n) (8) We have implemented the high-pass intra prediction approach in the framework of MC-EZBC software developed by RPI 5. In addition to the three modes known as DEFAULT (connected blocks), INTRA (unconnected blocks using backward MCP) and REVERSE (unconnected blocks using forward MCP), we define several intra-frame prediction modes. For the unconnected, i.e. INTRA and REVERSE blocks, a prediction block is formed based on the blocks preceding it in the scanning order. We have implemented a number of prediction directions based on the intra-prediction modes used by the MPEG-4 AVC/H.264 standard 1. The

4 LESZEK CIEPLINSKI, JORDI CABALL, SOROUSH GHANBARI experimental results reported below were obtained using the following set of modes: DC: the predictor is the average of the pixels from neighbouring blocks HORIZONTAL: prediction from the pixels from the block on the left VERTICAL: prediction from the pixels from the block above DIAGONAL_TOP_LEFT: prediction along the top-left diagonal DIAGONAL_TOP_RIGHT: prediction along the top-right diagonal We use the same intra prediction modes for luminance and chrominance components. The block-mode signalling is extended to allow for these additional modes. This leads to a slight overhead compared to the baseline codec, which is however more than offset by the fact that the intra-frame prediction modes do not require motion vectors to be transmitted. Mode selection is based on a comparison of the prediction error defined as Mean Absolute Difference (MAD) between the different intra prediction modes and the MC prediction. More specifically, the MAD incurred by using various intra prediction modes and motion-compensated prediction are compared and the mode with the lowest MAD is selected. An interesting problem with intra-frame prediction for scalable coding is that the decoder uses a different reference block than the encoder when forming prediction. One approach to solving this issue has been proposed in Wu 6, where authors suggest modified quantisation of intra-predicted blocks to account for the difference but do not provide a practical implementation. Here we propose a simpler approach that reduces the most drastic manifestations of the problem, which occur when intra prediction is repeated many times. To avoid this, we ensure that the intra-predicted blocks are not used for prediction of subsequent blocks. Improved prediction performance can be obtained if predictions from blocks that follow the current block in the scanning order are allowed. We have implemented a version of the codec using the following additional prediction directions HORIZONTAL_RIGHT, VERTICAL_BOTTOM, DIAGONAL_BOTTOM_RIGHT and DIAGONAL_BOTTOM_LEFT. While the additional modes increase the overhead in the block mode bits, we have found that this is outweighed by the improved prediction resulting from the additional flexibility. 3. RESULTS Figure 1 shows the results obtained on a subset of test sequences defined by MPEG for use in Exploration Experiments. More results can be found in Cieplinski 7, however a slightly different implementation was used there.

5 Intra-frame prediction for high-pass frames in motion compensated 5 wavelet video coding PSNR (db) Stefan Stefan-Intra Bus Bus-Intra bitrate (kbps) Figure 1. Performance for causal prediction directions It is seen that the performance generally improves when intra prediction is used, although the results are highly dependent on the sequence. The improvement is particularly significant for sequences with a high amount of complex motion, where motion estimation is likely to fail, and at low bitrates, where motion information constitutes a significant proportion of the overall bit budget. Table 1 presents the improvement that can be obtained by introducing the non-causal prediction modes. Table 1. Performance for non-causal prediction directions Bit Rate (kbps) Stefan-intra Stefan-intra + non-causal Bus-intra Bus-intra + non-causal It is seen that some improvement is obtained for the Stefan sequence, but it is not clear that it is sufficient to justify the additional processing complexity. Figure 2 shows a visual comparison of the intra-predicted and baseline codec for the Stefan sequence encoded at 256 kbps. It is seen that the visual improvement is quite significant.

6 LESZEK CIEPLINSKI, JORDI CABALL, SOROUSH GHANBARI Figure 2. Performance of Stefan sequence with baseline (left) and intra-predicted (Right) encoded at 256 kbps 4. CONCLUSIONS We presented results of intra-frame prediction for blocks in high-pass frames, which cannot be effectively coded using motion-compensated filtering. While the results obtained so far are encouraging, we believe further improvement in prediction quality is possible. One interesting direction is extending the prediction modes to include interpolation in cases where it is possible (see Wu 6 ). Other topics of interest are further limiting the impact of error propagation from multiple predictions within a picture and reduction of visual artifacts, particularly for diagonal prediction directions. 5. REFERENCES 1. ISO/IEC :2003: Information technology Coding of audio-visual objects Part 10: Advanced video coding, Ohm, J. R., Three-dimensional subband coding with motion compensation, IEEE Trans. Image Processing, 3:5: Hsiang, S. T., and Woods, J., 2001, Embedded video coding using invertible motion compensated 3D subband/wavelet filter bank, Signal Proc. Image Comm., 16: Taubman, D., and Secker, A., 2001, Motion-compensated highly scalable video compression using adaptive 3D wavelet transform based on lifting, Proc. IEEE Int. Conf. Image Proc., MC-EZBC software package, ftp://ftp.cipr.rpi.edu/personal/chen/. 6. Wu, Y., and Woods, J., 2003, Recent improvements in the MC-EZBC video coder, MPEG contribution M10396, Hawaii, USA, 7. Cieplinski, L., Caball, J., and Secker, A., 2003, Intra-frame prediction of high-pass frames in MCTF, MPEG contribution M10142, Brisbane, Australia