Software-embedded data retrieval and error concealment scheme for MPEG-2 video sequences Corinne Le Buhan Signal Processing Laboratory Swiss Federal Institute of Technology 1015 Lausanne - Switzerland ABSTRACT This paper considers the problem of data recovery and reconstruction in erroneous MPEG-2 video sequences. The basic resynchronization point in MPEG-2 video bitstream is the slice header, a slice usually denoting a full row of macroblocks. When an error occurs, the rest of the damaged slice is lost up to the next slice. In order to improve the efficiency of conventional error concealment schemes, we propose to exploit the error detection information to force the decoding of errorfree bits immediately after a lost area, before reaching either the next resynchronization point or the next erroneous area. This early resynchronization is achieved by trying to decode variable length codes until some macroblocks are recognized. In order to retrieve differentially coded data such as DC coefficients and macroblocks positions, a specific algorithm has been designed which uses not only available AC coefficients and neighbouring data, but also differential values decoded from the earlyresynchronized bitstream. This algorithm has been embedded in an MPEG-2 software decoder, combined with classical spatial and temporal error concealment techniques to interpolate the remaining lost areas. Simulation results show that up to 70% of the lost macroblocks can be retrieved in an intraframe coded picture. This retrieval along with temporal propagation yields a gain of several dbs as well as a visual enhancement over the whole sequence. Keywords: error concealment - MPEG-2 video sequences - resynchronization - interpolation 1. INTRODUCTION The work presented in this paper has been performed in the framework of the European Race project dttb (*). In this application, video sequences are coded according to MPEG-2 standard and broadcast through a terrestrial channel. Beside the use of channel error resilience techniques, single as well as burst errors may remain in the received bitstream. Because of data redundancy reduction due to compression, even a single bit loss may cause large visible errors in the decoded images. These errors propagate in both spatial and temporal directions because of the use of prediction and differential coding. Therefore, a robust error concealment scheme must be embedded in the MPEG-2 video decoder in order to deal with erroneous bitstreams by concealing residual errors at the image level. In the first part of this paper, we describe the source coding, channel simulation and basic error concealment techniques we used; we point out the effect of temporal propagation of imperfect concealment on the results. In the second part, we propose to force the decoding of the bitstream as soon as it is signaled error-free by a strict error detection scheme, before reaching the next resynchronization point, in order to use as much as possible of error free data. This early resynchronization requires the retrieval of differentially coded data; thus we describe a new concealment algorithm dedicated to this problem. Our algorithm has been integrated in a software MPEG-2 decoder and we provide PSNR results as well as images for visual evaluation in the third part. 2. PERFORMANCE OF ERROR CONCEALMENT IN MPEG-2 FRAMEWORK 2.1 Erroneous MPEG-2 bitstreams Digital television coding is now mainly based on MPEG-2 standard [1]. In our study, we consider the standard TV layer: 720*576 4:2:0 pictures, 25 images per second, compressed at 5Mbit/s. MPEG-2 video sequences are divided into groups of pictures (usually half a second long), starting with one intra coded I-picture followed by temporally predicted P- and B-pictures.
Each picture is composed of slices: a full row of macroblocks (16 lines) in MPEG-2 test models. Macroblocks (16*16 pixels) are composed of 4 luminance (Y) and 2 chrominance (U,V) 8*8 blocks which are DCT-transformed. In MPEG-2 video bitstream, a startcode and a header are associated with each sequence, each group of pictures, each picture, and each slice. No startcode is associated with macroblocks and blocks: when an error occurs within a slice, the decoder normally skips the remaining macroblocks until it reaches the next startcode at the beginning of the following slice (16 lines). For resynchronization purpose, no reference is made from one slice to the previous one, but spatial redundancy is reduced within the slice itself by means of differential coding of macroblock addresses, DC coefficients and motion vectors. Thus the difference between each DC coefficient and the previous one in the slice is encoded in the bitstream instead of the coefficient itself. The same technique is applied to motion vectors in P- and B-pictures. AC coefficients are run-length coded independently from neighbouring blocks. Variable length codewords are used in order to increase the compression ratio. In dttb application, video sequences are MPEG-2-encoded and combined with other streams to build the MPEG-2 transport stream as specified in MPEG-2 system specification [1]. This stream is packetized into 188 bytes long packets which are channel coded, broadcast, and channel decoded. The channel model [2,3] designed by DLR (**) in dttb project is used to affect the bitstream by introducing burst and single bit errors at various bit error rates (BER). At the receiver side, the video bitstream is reconstructed and entered to the source decoder. The MPEG-2 decoder is able to detect some errors, for instance when the decoded bits do not correspond to any expected codeword. However, no error can be detected when the erroneous bits map the initial codeword into another one. For each packet in the transport stream, one bit in the four bytes header is dedicated to error detection; this bit is handled by the channel simulator in our application. Such an information is also available in ATM applications (lost data packets). The MPEG-2 decoder can be extended with channel error detection capability: as soon as some bits are read from the bitstream, the decoder checks whether they belong to a corrupted packet. The whole packet is then skipped and the resynchronization procedure (startcode search) is applied at the next error-free area in the bitstream. This error detection scheme insures that no erroneous bits will be decoded. 2.2 Performance of error concealment techniques The skipped erroneous areas must then be concealed. Various error concealment techniques are available [3,4,5,6]. In our study, we have considered the algorithms proposed by DLR in dttb framework: spatial and temporal interpolation. Temporal redundancy can be exploited in video sequences by concealing lost areas with the corresponding areas from a previous picture. Simple temporal concealment is usually applied de facto in MPEG-2 decoders: lost macroblocks are replaced with the ones at the same position in the reference frame buffer containing the previous I- or P- picture used for prediction. The main drawback is that blocking effects appear in fast moving sequences, because lost macroblocks are not replaced with the matching ones. In P- and B-pictures, motion compensation can enhance temporal concealment: lost macroblocks are replaced with the matching macroblock in the previous reference picture. To achieve motion compensation, lost motion vectors are approximated by the median of their spatial surroundings. This technique outperforms simple replacement [3] (Fig.1a). Temporal error concealment cannot be applied at scene changes. Only spatial redundancy can be exploited then. We have used the spatial interpolation algorithm adapted to MPEG-2 features presented in [4] : lost macroblocks are linearly interpolated from available boundary pixels. This spatial error concealment technique suffers from the lack of neighbouring information, and its low-pass feature causes some blurring. When available, motion compensated error concealment yields better results in terms of both subjective evaluation and PSNR measures (Fig.1a). In most cases, interpolated information is poorer than the original lost one. When error concealment techniques are embedded in an MPEG-2 decoder, imperfectly concealed areas are used in the decoding process as well as for further concealment. This degrades image quality for the whole group of pictures. To emphasize the effect of temporal propagation, we have distinguished in our measures between absolute and embedded evaluation of error concealment techniques. Absolute evaluation is achieved by decoding and concealing the pictures with error-free reference frames; this scheme is used to study and compare the error concealment techniques in an absolute way. Then the algorithms are integrated in the MPEG-2 decoder so that error concealment imperfections are propagated in the decoding process. The first picture must be spatially concealed; next I-pictures can be either spatially interpolated or simply temporally replaced, and all three techniques can be applied to P- and B-pictures. Temporal propagation (Fig.1b) is responsible for several dbs of PSNR loss in error detection ('no concealment' curves: effect of error propagation in the decoding process) as well as in error concealment results (motion compensated temporal concealment 'motion', simple temporal replacement 'simple' and spatial interpolation 'spatial': effect of imperfect concealment propagation).
35 Absolute evaluation 35 Embedded evaluation 30 no error 30 no error PSNR (db) 25 20 15 simple spatial motion no concealment PSNR (db) 25 20 15 motion spatial simple 10 10 no concealment 5 0 5 10 15 Frame 5 0 5 10 15 Frame Fig.1. Comparison of error concealment techniques a) absolute evaluation b) embedded evaluation Therefore the concealment of reference I- and P-frames, used for temporal prediction, is really important: as much as possible of error free data in the bitstream should be used to limit the size of areas to conceal in these frames. Next part describes an algorithm allowing error free macroblocks retrieval in intra coded pictures. This algorithm is based on early resynchronization combined with the concealment of differentially coded data. 3. EARLY INTRA-SLICE RESYNCHRONIZATION AND MACROBLOCK RETRIEVAL 3.1 Forced resynchronization on error free macroblocks The slice structure of MPEG-2 test models does not facilitate efficient error concealment [7]. Indeed, usual resynchronization as defined in MPEG-2 scope proposes that the decoder looks for a startcode, appearing every 16 lines in the picture. This causes the loss of adjacent macroblocks in the horizontal direction, but also possibly in the vertical direction when consecutive slices are damaged or when the skipped erroneous packet contains a slice header. Error concealment cannot be efficient when the surrounding macroblocks are lost: no motion vectors can be found for motion compensated temporal concealment, and no pixels are available at the boundaries for spatial interpolation. To deal with the slice loss problem while keeping MPEG-2 syntax, S.Lee et al. [8] proposed to retrieve a large non-erroneous part of the slices by means of a forced resynchronization on VLC codewords corresponding to macroblocks data. The decoder is forced to decode macroblocks from the bitstream as soon as it is detected error-free, starting at the first bit after a lost packet and decoding either until an invalid codeword is detected, which means the resynchronization did not start at the right place (iterate the process at next bit then), or until the whole retrieved bitstream is successfully decoded (next startcode or next erroneous data packet reached). In the following, we call 'subslice' the resulting early-resynchronized macroblocks. 3.2 Recovery of differentially coded data Once macroblocks are retrieved, they must be positioned in the picture and the associated differentially coded data must be concealed. When macroblocks are retrieved between a lost slice area and the next slice startcode, there is no uncertainty on their position: they are at the end of the damaged slice (backward resynchronization). However, the retrieved macroblocks positions cannot be easily known when they are delimited by erroneous areas. Indeed, the use of variable length codes does not allow to know how many macroblocks are damaged in a lost packet. The worst case is when slice headers are lost: the uncertainty over
the position can cover several slices in this case. Therefore, a positioning technique must be designed to locate the retrieved data between erroneous areas. Moreover, the DC coefficient for each block is differentially coded by means of a simple DPCM coding technique. The coded data is the difference between the previous and the current DC value in the slice. When a macroblock is lost, the DC predictor is lost; for the next retrieved macroblocks, DC coefficients must be concealed. I-pictures are used as temporal reference for a whole group of pictures, so they must be well concealed although less error concealment techniques are available in their case. No macroblocks can be skipped in I-pictures: the macroblock address increment is always equal to one. P-pictures are also used as temporal reference, but motion compensated error concealment applies to them; moreover, the motion vectors themselves and the macroblock adresses are differentially coded, which makes the retrieval technique much more complex than in I-pictures. For these reasons, we have limited macroblock retrieval to I- pictures. 3.3 DC coefficient interpolation We have first investigated DC coefficient interpolation for backward located macroblocks whose position at the end of the retrieved slices is known. Different strategies are possible to interpolate the DC coefficient from a macroblock when its neighbours are available. We have chosen not to exploit temporal redundancy so that the scheme can be applied to the first picture and above all in order to limit temporal propagation from one group of pictures to another. In the spatio-frequency domain, existing error concealment techniques such as the use of smoothing constraints [5] can be applied, but these schemes do not take into account differential values coded in the bitstream. Therefore we propose a new error concealment scheme which exploits spatial redundancy as well as available differential values. The basic principle consists of minimizing the difference between each lost block and its immediate vertical neighbour(s) from slices above and/or below (DC error, vertical spatiofrequency redundancy constraint). DC coefficients are also constrained by the decoded DC differential values (exact intra-slice constraint). The DC concealment error is minimized on the retrieved subslice to get corresponding initial DPCM-predictors (luminance and chrominance). Once these first DC values are concealed, DC coefficients in the next blocks are immediately derived, inverse DCT is performed (AC coefficients are directly available from the bitstream), and the resulting retrieved area is introduced in the picture. Luminance Chrominance (420) Upper neighbouring DC constraints slice DPCM difference values The following algorithm is applied to each chrominance block: Lower neighbouring DC constraints Fig.2. DPCM and neighbouring constraints Let DC(n) be the DC coefficient to conceal, for macroblock n, with n=0 to nb-1 where nb is the number of retrieved macroblocks in the subslice. Let DC A (n) and DC B (n) be respectively the upper and lower neighbour DC coefficients for macroblock n. For n>0, let n-1,n be the decoded DPCM value, and n be the cumulated decoded DPCM value: n-1,n = DC(n)-DC(n-1) (1) n = 0,n = n k 1,k k =1 Then we can write: DC(n) = DC(0) + n (2) (3)
As all the DC coefficients within the subslice are related to the first one by the available DPCM values, the interpolation error minimization problem can be expressed as a function of this first coefficient DC(0). We define the total interpolation error Err(DC(0)) to be minimized as a quadratic error: Err( DC(0)) = nb 1 n=0 Err(n) (4) where Err(n) =σ A (n)[ DC(0) + n DC A (n)] 2 +σ B (n)[ DC(0) + n DC B (n)] 2 (5) σ A (n) (resp. σ B (n)) = 1 if upper (resp. lower) neighbour is available, 0 otherwise (6) The error is minimum for derr(dc(0)) DC(0) = 0, which leads to: DC(0) = nb 1 n=0 { σ A (n)[ n DC A (n)]+σ B (n)[ n DC B (n)]} nb 1 n=0 { σ A (n) +σ B (n)} (7) Once the first DC coefficient DC(0) is concealed, the others are given by the differential values (Eq.3). This algorithm directly applies to chrominance blocks in 4:2:0 format. For luminance blocks, the DPCM scheme is a bit more complicated because there are four blocks in the macroblock. Basically, the same algorithm can be applied by taking into account the DPCM scheme (Fig.2) to calculate n where n is up to 4*nb-1 instead of nb-1. This way at most one neighbour is available: the upper one for top blocks and the lower one for bottom blocks, as represented on figure 16. Once DC coefficients are concealed, inverse DCT is simply performed. 3.4 Retrieved macroblocks positioning The position of early-resynchronized macroblocks between damaged areas must also be retrieved (Fig.3). Basically, we propose to investigate each possible subslice position in the slice, apply the above DC concealment technique, and keep the position with minimal DC error. This technique may be fooled in some cases, depending on the neighbouring slices structure. Therefore we propose to also take into account vertical mapping of boundary pixels, which corresponds to a local constraint on vertical contours. The retrieved position is the one that minimizes the joint DC/boundary error. When not enough neighbours are available (adjacent damaged slices), the retrieval process must be iterated, taking into account previously retrieved areas. A control scheme should also be implemented to eliminate wrong DC concealment (resp. positioning). The decision to whether retrieved macroblocks can be introduced in the final picture or not is based on DC concealment (resp. joint DC-boundary) error thresholding. Furthermore, a complementary decision scheme must insure the elimination of false resynchronization due to the decoding of shorter or larger than original variable length codes. We have observed that the resulting false macroblocks are usually limited to the first ones in the retrieved subslice, before the decoder is correctly resynchronized on variable length codes.? erroneous macroblocks slice startcode retrieved macroblocks retrieved macroblocks backward resynchronized error-free macroblocks?? backward resynchronized error-free macroblocks Fig.3. Macroblock positioning problem
Therefore, once macroblocks are positioned and DC-concealed, the individual joint DC-boundary error is checked for each macroblock from left to right until it is below a fixed threshold. The corresponding macroblocks are eliminated and the DC concealment/macroblocks positioning scheme is iterated on the reduced subslice to avoid taking into account false data. 4. EXPERIMENTAL RESULTS We have embedded our algorithm in an MPEG-2 software decoder and we have evaluated it on several video sequences at bit error rates (BER) of 2.10-3 and 2.10-4 [9]. In most I-pictures, the remaining lost areas may be either spatially interpolated (Fig.10) or temporally replaced (Fig.8), but the former technique provides better results in fast moving sequences while avoiding error propagation from one group of pictures to another. Predicted pictures are concealed with the motion compensation scheme. Fig.4 illustrates the resulting gain of PSNR over usual resynchronization schemes for 'Flower Garden' at BER=2.10-3 (Fig.4a) and BER=2.10-4 (Fig.4b): the size of areas to be concealed is strongly reduced, more neighbours are available, so spatial interpolation becomes more efficient. This improved concealment propagates over the next predicted pictures (Fig.4a). Up to 70% of usually lost macroblocks can be retrieved, DC concealed and located well enough to be introduced in the final I-picture. Fig.7 compared to Fig.6 illustrates the proportion of retrieved macroblocks in the intra-coded 13 th frame of sequence 'Flower Garden' at BER=2.10-3 : about 15% of the total number of macroblocks are retrieved by means of early resynchronization. Final concealed pictures show the visual enhancement brought by the technique (Fig.8,9,10). Flower Garden, 20 frames, BER=0.002 35 Flower Garden, 125 frames, BER=0.0002 35 30 no error 25 30 no error 20 early early 15 usual 25 usual 10 20 5 no concealment 0 0 5 10 15 Frame no concealment 15 0 50 100 Frame Fig.4. Comparison between early and usual resynchronization schemes a) BER=2.10-3, 20 frames b) BER=2.10-4, 125 frames
Fig.5. Error free decoded I-picture Fig.6. Usual resynchronization Fig.7. Early resynchronization, macroblock retrieval in damaged slices Fig.8. Early resynchronization, simple temporal replacement Fig.9. Usual resynchronization, spatial concealment Fig.10. Early resynchronization, spatial concealment
5. CONCLUSION In this paper, we have dealt with the problem of data recovery in MPEG-2 erroneous video sequences. It was shown how temporal propagation of imperfect error concealment affects the performance of MPEG-2-embedded error concealment techniques. Therefore we propose to reduce the size of areas to conceal in I-pictures, used as temporal reference for a whole group of pictures. This is achieved by means of an early resynchronization on variable length codes in the MPEG-2 decoder, combined with a macroblock positioning algorithm and a concealment method for differentially coded DC coefficients. The proposed technique has been embedded in a software MPEG-2 decoder in combination with spatial interpolation and motion compensated error concealment methods. Yielding the recovery of up to 70% of the lost macroblocks, our algorithm has been proven to be more efficient than usual resynchronization based schemes at bit error rates of up to 2.10-3. Possible extensions and enhancements include the use of an adaptive spatio-temporal error concealment technique [6] ; smoothing constraints based algorithms [5] may also be extended to take into account available DPCM-coded data as an additional constraint. 6. ACKNOWLEDGEMENTS The author would like to thank S.Aign, K.Fazel and P.Robertson from DLR (**) for providing her with the channel model, useful advice and technical hints. (*) dttb: European Race project 2082, "Digital Terrestrial Television Broadcast" (**) DLR: German Aerospace Research Establishment - Institute for Communications Technology 7. REFERENCES [1] MPEG-2 Draft International Standard H.262, ISO/IEC 13818-1 and 13818-2, May 1994 [2] K.Fazel et al, "Interface source channel coding and BER management", Sept.1994, European Race project dttb, report dttb/m3/196 [3] S.Aign and K.Fazel, "Error detection and concealment measures in MPEG-2 video decoder",proceedings of the Workshop on HDTV'94, Turin, Italy, Oct.1994 [4] S.Aign and K.Fazel, "Temporal and spatial error concealment techniques for hierarchical MPEG-2 video codec", IEEE International Conference on Communications ICC'95, vol.3, pp.1778-1783, Seattle, June 1995 [5] Q.F.Zhu, Y.Wang, L.Show, "Coding and cell-loss recovery in DCT-based packet video", IEEE Transactions on Circuit Systems and Video Technologies, vol.3(3), pp.248-258, 1993 [6] H.Sun, J.Zdepski, "Adaptive error concealment algorithm for MPEG compressed video", Visual Communications and Image Processing'92, Proceedings of SPIE, vol.1818, pp.814-824, Boston, 1992 [7] Y.Q.Zhang, X.Lee, "Performance of MPEG codecs in the presence of errors", Journal of Visual Communications and Image Representation, vol.5(4), pp.379-387, Dec.1994 [8] S.Lee et al, "Transmission Error Detection, Resynchronization, and Error concealment for MPEG Video Decoder", SPIE'93, vol.2094, pp.195-204, 1993 [9] C.Le Buhan, "Error Concealment based on early resynchronization in MPEG-2 video sequences", Technical Report LTS/EPFL no.95-06, June 1995