3D: How Video Compression Technology can contribute

Transcription

1 SMPTE Meeting Presentation Pierre Larbier, CTO 3D: How Video Compression Technology can contribute ATEME, Bièvres France, Written for presentation at the [SMPTE International Conference on Stereoscopic 3D for Media and Entertainment] Abstract. Transmission of 3D TV content has become a reality in 2010 with DTH services operators committing to commercial deployments. To avoid video encoders and set-top boxes modifications, high definition content is constructed by packing two views onto a single video stream. For contribution, removing information from the original views could introduce dramatic degradation to the edited and distributed 3D video. This can be avoided by transmitting independently the two views (simulcast) with standard contribution encoders and decoders. This paper will present the quality tradeoffs of the most popular 3D packing schemes from a video compression perspective with particular attention paid to the specific AVC/H.264 artifacts that they might cause. We will also present deployed innovative AVC/H.264 4:2:2 10-bit compression solutions that overcome quality degradation in the 3D contribution space with special attention paid to simulcast synchronization issues. Keywords. 3D,H.264,AVC,contribution,sub-sampling The authors are solely responsible for the content of this technical presentation. The technical presentation does not necessarily reflect the official position of the Society of Motion Picture and Television Engineers (SMPTE), and its printing and distribution does not constitute an endorsement of views which may be expressed. This technical presentation is subject to a formal peer-review process by the SMPTE Board of Editors, upon completion of the conference. Citation of this work should state that it is a SMPTE meeting paper. EXAMPLE: Author's Last Name, Initials Title of Presentation, Meeting name and location.: SMPTE. For information about securing permission to reprint or reproduce a technical presentation, please contact SMPTE at jwelch@smpte.org or (3 Barker Ave., White Plains, NY 10601).

2 Introduction Broadcasters are considering using a phased approach for Stereoscopic 3-D (S3D) delivery to the home. The first phase, often described as Frame Compatible or Phase 1, enables the existing HD infrastructure to carry S3D up to the user s new 3D TV screen. This first phase has already started as numerous broadcasting experimentations have been performed since the end of last year. Regular programs have even been launched recently. A second phase, called Service Compatible, will allow the use of new techniques to improve delivery efficiency and user experience. It is usually considered that Phase 2 could be launched in the next three years. The main rationale for the Frame Compatible phase is to avoid replacing the large number of set-top-boxes already deployed. Therefore the video format decoded by the set-to-boxes shall be the same as it is today: 1080i or 720p compressed in AVC/H.264 (or possibly MPEG-2) and encapsulated in an MPEG-2 Transport Stream. Consequently, the delivery infrastructures, that include video encoders, remain mostly identical. The scheme devised to keep the current video formats is to pack the two views that compose an S3D signal onto a single 1080i or 720p video stream. Consequently, each view has to fit in half of a frame size. This requires a form of down-sampling or simplification of the pictures prior to packing, to achieve a two-fold pixel count decrease of each view. And it remains then to be seen if the packing arrangement and/or the decimation algorithm affect the video quality output by the video encoders currently in use. In contrast, S3D Broadcast Contribution in Phase 1 does not have necessarily to obey to the same constraints than Distribution. The goal in this case is to maintain the highest possible video quality considering the transport means, even if it requires modifying encoders and decoders. Moreover, the Contribution S3D signal has to be compatible with the Distribution stream that is delivered to the home. Since various packing and down-sampling methods could be in use at the final delivery stage, Contribution scheme should be accommodating all of them. This paper will also present recent advances involving only minor encoder and decoder modifications and that fulfill all requirements of S3D Contribution applications, paving the way to large scale deployments. Frame Compatible formats Many methods have been proposed to simplify and pack the two views composing S3D content. But the recent publication of HDMI 1.4a has restricted significantly their number. Since this specification standardizes the communication link to the television set, only supported formats may be output by a set-top-box and displayed properly on the TV and it is very unlikely that other format emerge in the very near future. The HDMI 1.4a specification provides only two mandatory Frame Compatible formats for Broadcast content: Side-by-Side Horizontal for 1080i video sources Top-and-Bottom for 720p video sources 2

3 In addition, it describes secondary formats and future extensions that could also be used in Frame Compatible applications: Top-and-Bottom for 1080i video sources. Line Alternating for 1080i video sources sub-sampling for 1080i and 720p video sources This leaves three packing patterns and four down-sampling methods to construct a Frame Compatible video stream. The following table describes the possible combinations (in red the HDMI 1.4a mandatory formats): Packing Down-sampling Horizontal resize Vertical resize Side-by-Side Top-and-Bottom Line Alternating 1080i 720p 1080i 720p Line selection 1080i sub-sampling 1080i 720p The following figures describe the four possible combinations of packing and down-sampling: L R L R Side-by-Side with Horizontal Resize Top-and-Bottom with Vertical Resize L+R Line alternating Zoom view L R sub-sampling Zoom view 3

4 Side-By-Side with Horizontal Resize The left and right views are horizontally downscaled by a factor of two to produce a Side-by-Side video sequence that can be compressed and transmitted. Each view is resized to 960x1080 pixels when the source is 1080i (or in 720p) and then assembled to produce standard 1920x1080 (or 1280x720) frames. The important advantage of this scheme is that it does work equally well on interlaced and progressive content since there is no picture modification on the vertical axis. Several downscaling methods may be used. One common algorithm, relatively simple to implement is to use piecewise cubic filters (also known as Bicubic interpolation). Top-and-Bottom with Vertical Resize The left and right views are vertically downscaled by a factor of two to produce a Top-and-Bottom (also known as Over-Under) video sequence that can be compressed and transmitted. Each view is resized to 1920x540 pixels when the source is 1080i (or in 720p) and then assembled to produce standard 1920x1080 (or 1280x720) frames. The complexity of this scheme is the same as Side-by-Side with Horizontal Resize when the source is progressive (720p). However, great care must be taken to limit aliasing artifacts in the interlaced case (1080i). There are several methods to do this. We have chosen the following algorithm: De-interlace, downscale each now-progressive fields, then re-interlace. This technique produces very good results but could be awfully difficult to implement. In this paper, we have chosen to use this method using a state-of-the-art 5-field motion compensated de-interlacer. Line Alternating Each combined picture is constructed by selecting even lines from one view and odd lines from the other view. The result is an interlaced picture, even if the sources were progressive. This scheme is extremely simple to implement and was most probably designed to match polarized TV screens. Side-By-Side with Sub-Sampling To avoid down-sampling by a factor of two either horizontally or vertically, this scheme downsamples the source pictures by a factor of in each direction. The decimated views are then combined in a Side-by-Side pattern. This method has certain theoretical advantages. First of all, the definition loss is done symmetrically in both directions which could be less annoying for the human eye. Secondly, efficient reconstruction may be performed to reduce the visual degradation introduced by down-scaling. There are several algorithms to perform quincunx sub-sampling, some of them being patented. In the context of this paper, we have chosen to apply a large quincunx kernel (non-separable 12x12 taps rotated at 45 ) prior to diagonal sampli ng. When the content is potentially interlaced (1080i), this process is applied to both fields independently. 4

5 Performance evaluation of packing methods S3D perception is degraded if the two views exhibit a dissimilar video quality. Secondly, S3D compression may introduce a specific artifact compared to traditional 2D encoding: the reduction of stereoscopic perception on some part of the pictures. These imperfections can be obtained when there is some cross-talk between the two views. Consequently, the video encoder has to process identically and independently the two packed frames in order to avoid those defects and maximize the user experience. We have tried to verify the behavior of a current video encoder on the three possible packing arrangements. The idea is compare the rate-distortion curve of 2 video sequences: 1. A reference video sequence 2. A second sequence constructed by packing pictures from the reference sequence and simple transformations of these pictures that don t change their complexity If the curves are identical (i.e. the same quality is obtained for the same rate), then it means that the two packed frames were processed independently and identically. If not, then the encoder has managed to exploit redundancies between the two packed views (a picture and its transformation). This indicates that the packing arrangement has an influence on the encoder behavior, which in turn is the sign that specific degradations may arise. The video encoder configuration was chosen according to typical distribution use case: Video encoder: ATEME Kyrion encoder software model AVC/H.264 High Profile Rate-control mode: CBR Interlaced coding: Macrobloc-Adaptive Frame-Field (MBAFF) Hierarchical B-frames, GOP: 18 Many different sequences were tested and the figures presented hereafter show an example of the results: 37 Whale Show 1080i60 Side-by-Side 37 Whale Show 1080i60 Top-and-Bottom 37 Whale Show 1080i60 Line Alternating ) B32 (d R31 S N 30 P Reference Same Pictures Flip Vertical Flip Horizontal Reference Same Pictures Flip Vertical Flip Horizontal Reference Same Pictures Flip Vertical Flip Horizontal Side-by-Side and Top-and-Bottom packing arrangement exhibit an exact match between the various rate-distortion curves. This shows that the encoder does not in any way try to use the redundancies between the two packed views. We can then assume that the two views are processed independently with the same video quality. On the other hand, the curves obtained with the Line Alternating arrangement do not match. This is evidence that information from one field (one of the two views) is sometime used to help coding the other field (the other view). This behavior is expected from a video encoder when processing 2D interlaced content, especially when using the MBAFF tool. But it is clearly an issue in the context of S3D compression: The two views are not independently encoded and there is cross-talk between them. It is probably possible to optimize the encoder to avoid those 5

6 problems when using Line Alternating arrangement, for instance using exclusively field coding mode and avoiding cross-referencing fields. But this could decrease significantly the overall coding efficiency, which would in turn reduce the overall video quality. In addition, this kind of configuration is not available on most encoders currently in use. We can conclude from this evaluation that the packing methods used when transmitting S3D content in Frame Compatible mode have an impact when the video encoder is considered in the transmission chain. Side-by-Side and Top-and-Bottom provide the expected behavior whereas Line Alternating may introduce visual artifacts. It is also interesting to note that since the two tiles are processed independently with the Sideby-Side and Top-and-Bottom arrangements, the bit-rate needed for a S3D sequence is obtained at exactly twice the bit-rate needed for each view. Assuming for instance that 6Mbps provides a sufficient quality for a 2D HD sequence downscaled to 960x1080i; Then we can deduct that a 3D version of the same content will deliver the same quality to each eye at 12Mbps. Performance evaluation of down-sampling methods Most video encoders include a preprocessing stage to improve the visual perception, for instance to remove noise or simplify the video source. But a preprocessing stage not specifically designed to be combined with compression, such as S3D down-sampling, might in fact increase the video complexity. The consequence is a lower video quality which can be damageable to the user experience, especially considering very low bitrates allowed for delivery. So, after evaluating the influence of the packing arrangements, we have tried to measure the impact of the most common down-sampling methods. As indicated before, the HDMI 1.4a specification has standardized four down-sampling methods that can be used during the Frame Compatible phase: Horizontal down-scaling Vertical down-scaling Sub-Sampling Line Selection The Line Selection scheme is tied to Line Alternating packing arrangement, and we have seen that this packing method may by itself introduce annoying artifacts when the encoder is not specifically optimized to process this type of content. For this reason, we have not evaluated its impact on encoding. The HDMI 1.4a specification leaves the choice of down-sampling and up-sampling algorithms to equipment vendors. In order to avoid a bias toward a specific method in the course of this evaluation, we have chosen very similar high quality algorithms like the same interpolation kernel (16 taps with a Blackman window): As before, the video encoder configuration was also chosen according to standard distribution application. 6

7 Down-scaling 720p Two example rate-distortion curves between the source before down-scaling and the output after up-scaling are shown below. They describe the amount of information that is kept when considering the whole processing chain (down-sampling - encoding - decoding - up-sampling). ParkJoy 720p50 Dancer 720p They show that beyond approximately 15Mbps, the quincunx sub-sampling method keeps definitely more detail that the two other algorithms, which is confirmed by visual inspection. At 50Mbps, the difference between the three methods is significant: up to 3dB between the best and the worse method. Below this 15Mbps mark, horizontal or vertical sub-sampling takes the lead, but the difference is less important. Experiments with other content show that most of the times horizontal resizing provides better results in 720p than vertical resizing. But it seems to be content-dependant. For instance, the Dancer sequence tested here is highly contrasted and contains a lot of vertical edges that are smoothed out by the horizontal down-sampling. When looking at the video encoder internal decisions, it appears that quincunx sub-sampled pictures are more difficult to encode than horizontally or vertically resized pictures. For instance, the quantizer is always higher which means that less spatial or temporal redundancies are found in the down-scaled video. This is particularly obvious when looking at the PSNR ratings between the encoder input and decoder output as shown in the figures below. Horizontally and vertically resized pictures present almost the same complexity to the encoder, while quincunx sub-sampling is significantly more difficult to encode. Encoder only ParkJoy 720p50 Encoder only Dancer 720p The problem is that quincunxed sub-sampled pictures look as if there was a horizontal shift between adjacent lines. And this causes difficulties to the encoder motion estimation stage. 7

8 Furthermore, inter and intra predictions are slightly less efficient. The net effect is a loss of coding efficiency and a slight raise of the quantizer to keep the required bit-rate. Unfortunately, the consequence of a higher quantizer is an increase in artifacts caused by the encoder itself (blocs, blurriness, smearing etc.) However, at Distribution bitrates, between 3Mbps and 8Mbps for each view, it is difficult to choose between the three down-sampling methods since they provide similar results after compression: Objective measurements are close Horizontal and vertical resizing may appear blurry sub-sampling might provide more details but at the expense of stronger encoding artifacts Down-scaling 1080i Sub-sampling 1080i content is much more complex than 720p since the video is transmitted in two separate fields, even if it is progressive. Efficient vertical and quincunx sub-sampling is difficult to perform since they have to be done across both fields. The following figures present some results obtained with typical Broadcast content. NouvelleStar 1080i50 Wanted 1080i x x x x540 These curves show clearly that the horizontal down-scaling keeps the most details, whatever the bit-rate. sub-sampling comes in second while vertical scaling gives poor results. The difference between the three methods is very significant above 10Mbps for each view, and can as high as 5dB at 50Mbps. Furthermore, this ranking was observed on all tested contents and was confirmed by a visual evaluation. As in 720p, internal encoder metrics show that quincunx sub-sampling is more complex to compress than horizontal or vertical resize. But the main problem in 1080i is caused by the vertical processing performed on interlaced content. Beyond the Frame Compatible Phase Toward Phase 1.5 The main idea behind the Frame Compatible phase is to avoid replacing the large number of set-top-box currently in use. This leads to pack the two views onto a single standard frame. 8

9 This can be a serious constraint with for instance only 360 lines videos in 720p Top-and-Bottom mode; quite far from High Definition. Another option could be to adapt S3D streams to exploit the maximum capabilities of existing set-top-boxes. In this case, the limit is not the picture size but codec level. Among many parameters, the level indicates the maximum number of macroblocs that a decoder can handle per second. And today s AVC/H.264 decoding devices are specified to handle at least level 4.0, which can accommodate frames made of 8192 macroblocs at 30Hz. A compliant AVC/H.264 decoder should be able to process smaller frames at a higher frame rate provided this limit is not exceeded. To avoid the definition loss either horizontally or vertically, quincunx sub-sampling has been proposed. But we have seen that a standard AVC/H.264 encoder is not comfortable with this scheme at low bitrates. Another option could then be to use a simple orthogonal sub-sampling but at twice the frame rate and interleave temporally the two views. For instance, 1080i pictures could be downscaled to 1344x768i and 720p pictures could be downscaled to 960x544p. Larger frame size could even be used for 25Hz videos like 1440x864i or 1024x604p. Furthermore, interleaving the two views temporally instead of spatially enables to use MVC (Multi-View Coding) that could improves the coding efficiency in a compatible way by about 15%. The advantages of this scheme are obvious, especially in 720p where the definition loss would be moderate compared to 2D. However, it assumes that the decoding device includes a flexible upscaler capable of handling non-standard frame sizes. It could be the case for the most advanced decoding chips but not guaranteed for legacy decoders. We have tested this new method on several sequences, and it appears to outperform all Frame Compatible sub-sampling methods, not counting the potential gain of using MVC. For instance, the following figures give the results obtained with the Dancer sequence. Dancer 720p50 Dancer 720p50 960x544 Mean Quantizer x How about even larger frame sizes? Having shown that larger frame size might give better video quality at all useable bit-rates, we went a step further and tested customary used 2D resolutions up to full-hd. As exemplified in the following figures, increasing the frame size also improves the video quality. If the available bandwidth exceeds 20 Mbps (total for both views), keeping the full resolution will almost always deliver the best video quality. This result is well known in standard 2D video broadcasting. There is a tradeoff between resolution and bitrate to achieve a given quality level and the bitrate has to be very low to justify 9

10 using half-resolution. Since the delivery rate envisioned for S3D distribution is around 12Mbps or more, the best configuration would have been to use ¾ resolution for each view. Unfortunately, this is not yet compatible with most decoding devices. Tennis 2x1080i50 (3D) BMX 2x1080i60 (3D) x960x1080 2x1920x1080 2x1440x1080 2x960x1080 2x1920x1080 2x1440x D Contribution We have seen that above approximately 20Mbps, down-sampling the two views of an S3D signal will reduces the video quality. Since Contribution is usually carried out above this rate, it is a natural inclination to transmit full-hd views for Contribution applications. Besides, it solves a major problem that may arise when the stream is re-distributed by different networks: Frame Compatible formats cascading may cause catastrophic definition losses (for instance Top-and- Bottom followed by Side-by-Side). But without actual 3-D encoder and decoders, the deployment of such a solution is very challenging. Since the end of last year, ATEME has made several experiments with Broadcasters and has developed a solution using off-the-shelf Contribution encoders and receivers. It culminated last May with a first satellite Contribution of a popular commercial program produced in 3D with 3 redistribution networks able to reach thousands of subscribers. The delivery to the home was made using Frame Compatible equipments, but the primary Contribution used a simulcast technology. The following figure depicts the set-up used for this event. Right View Left View Contribution Encoders 2x CM4101 Multiplexer MC3100 3D MPTS 3D MPTS Contribution Decoders 2x DR Optional Genlock source Right View Left View 3D Distribution 10

11 The two views are encoded independently by two un-modified AVC/H.264 4:2:2 10-bit Contribution encoders. The resulting Single Program Transport Streams (SPTS) are then multiplexed together to be transmitted. At the reception side, the stream is sent to two Contribution receivers that have received a small software modification to enable perfect synchronization (patent pending). The output of the receivers can then be edited and delivered. Achieving synchronization relies in fact on the properties of a compliant MPEG-2 Transport Stream. The two receivers have to select the same PCR clock to reconstruct the video clock to achieve synchronization. And this is possible because the decoders receive both streams multiplexed in an MPTS. The synchronization accuracy achieved with this method is better than 100ns. This means that both receivers are able to output the two views synchronously, without any physical link between them. Conclusion The HDMI consortium has issued in March 2010 the 1.4a specification that standardizes de-facto the useable video formats for the first phase of stereoscopic 3-D deployments. Two frame arrangements are mandatory for the newly available 3-D TVs: Side-by-Side for 1080i content and Top-and-Bottom for 720p content. However, other formats are available as options. When considering the compression stage and the bitrates used in the delivery chain, it appears that the two mandatory formats offer the best video quality. Other optional packing arrangements or sub-sampling methods might sometimes provide better intermediary results at the expense of a significant increase in complexity for the encoder. But other methods can be envisioned if the constraints of the Frame Compatible phase do not apply. It is for instance the case with Broadcast Contribution where the stream is not decoded by set-top-boxes, but by professional-grade synchronizable receivers. In this case the degradation introduced by down-sampling can be avoided altogether, allowing transmission of 3-D signals with the highest possible video quality. Only recently, it was possible to perform high quality primary Contribution of commercial 3-D program via satellite toward several Distribution networks. This was done by packing in a single MPTS the two views, compressed independently by unmodified AVC/H.264 4:2:2 10-bit encoders. At the reception points, pairs of synchronized receivers were able to recover the original stereoscopic 3D program without any visible artifact. Acknowledgements Thanks to Adi Kouadio from the European Broadcasting Union and Ludovic Noblet and Christophe Daguet from Orange Labs for their help during the preparation of this paper. References 1. HDMI LLC, High-Definition Multimedia Interface Specification Version 1.4a, Extraction of 3D Signaling Portion, March 4, 2010 (freely available at 2. ISO/IEC :2009 Information technology Coding of audio-visual objects Advanced Video Coding» 11