Overview: Video Coding Standards

Transcription

1 Overview: Video Coding Standards Video coding standards: applications and common structure Relevant standards organizations ITU-T Rec. H.261 ITU-T Rec. H.263 ISO/IEC MPEG-1 ISO/IEC MPEG-2 ISO/IEC MPEG-4 Recent progress: H.264/AVC Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 1

2 The JVT Project ITU-T Q.6/SG16 (VCEG - Video Coding Experts Group) formed for ITU-T standardization activity for video compression since 1997 August 1999: 1st test model (TML-1) of H.26L December 2001: Formation of the Joint Video Team (JVT) between VCEG and ISO/IEC JTC 1/SC 29/WG 11 (MPEG) to establish a joint standard project - H.264 / MPEG4-AVC ITU-T Approval: May 2003 ISO/IEC Approval: October 2003 Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 2

3 JVT Goals Improved coding efficiency Average bit rate reduction of 50% given fixed fidelity compared to any other standard Trade-off complexity vs. coding efficiency Improved network friendliness Anticipate error-prone transport over mobile networks and the wired and wireless Internet Further improve robustness techniques in H.263 and MPEG-4 Simple syntax specification Avoid excessive quantity of optional features Minimize number of profiles for distinct application areas Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 3

4 H.264/JVT Applications Entertainment Video Broadcast: Terrestial / Satellite / Cable... Storage: DVD / HD-DVD / PVR... Conversational Services H.320 Conversational 3GPP Conversational H.324/M H.323 Conversational Internet/best effort IP/RTP 3GPP Conversational IP/RTP/SIP Video Streaming 3GPP Streaming IP/RTP/RTSP Streaming IP/RTP/RTSP (without TCP fallback) Other Applications 3GPP Multimedia Messaging Services Digital camcorder Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 4

5 Relationship to Other Standards Identical specifications have been approved in both ITU-T / VCEG and ISO/IEC / MPEG In ITU-T / VCEG this is a new & separate standard ITU-T Recommendation H.264 ITU-T Systems (H.32x) will be modified to support it In ISO/IEC / MPEG this is a new part in the MPEG-4 suite Separate codec design from prior MPEG-4 visual New Part 10 called Advanced Video Coding (AVC similar to AAC in MPEG-2 as separate audio codec) MPEG-4 Systems / File Format has been modified to support it H MPEG-2 Systems also modified to support it IETF: RTP payload packetization Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 5

6 H.264/AVC Profiles Baseline: core compression capabilities, plus error resilience, e.g., for videoconferencing, mobile video Main: high compression and quality, e.g., for broadcasting Extended: added features for efficient streaming Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 6

7 H.264/AVC Coder Input Video Signal Split into Macroblocks 16x16 pixels - Decoder Coder Control Transform/ Scal./Quant. Scaling & Inv. Transform Control Data Quant. Transf. coeffs Entropy Coding Intra/Inter Intra-frame Prediction Motion- Compensation Deblocking Filter Output Video Signal Motion Estimation Motion Data Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 7

8 Input Video Signal Progressive Frame Top Field Bottom Field Progressive and interlaced frames can be coded as one unit Progressive vs. interlace frame is signaled but has no impact on decoding Each field can be coded separately Δt Dangling fields Interlaced Frame (Top Field First) Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 8

9 Partitioning of the Picture Slices: A picture is split into 1 or several slices Slices are self-contained Slices are a sequence of macroblocks Macroblocks: Basic syntax & processing unit Contains 16x16 luma samples and 2 x 8x8 chroma samples Macroblocks within a slice depend on each other Macroblocks can be further partitioned Slice #0 Slice #1 Slice # Macroblock #40 Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 9

10 Flexible Macroblock Ordering (FMO) Slice Group: Pattern of macroblocks defined by a Macroblock allocation map A slice group may contain 1 to several slices Macroblock allocation map types: Interleaved slices Dispersed macroblock allocation Explicitly assign a slice group to each macroblock location in raster scan order One or more foreground slice groups and a leftover slice group Slice Group #0 Slice Group #0 Slice Group #1 Slice Group #2 Slice Group #0 Slice Group #1 Slice Group #1 Slice Group #2 Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 10

11 Interlaced Processing Field coding: each field is coded as a separate picture using fields for motion compensation Frame coding: Type 1: the complete frame is coded as a separate picture Type 2: the frame is scanned as macroblock pairs, for each macroblock pair: switch between frame and field coding Macroblock Pair Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 11

12 Scanning of a Macroblock 0 1 Intra_16x16 macroblock type only: Luma 4x4 DC Coded Block Pattern for Luma in 8x8 block order: signals which of the 8x blocks contains at least one 4x4 block with nonzero transform coefficients Luma 4x4 block order for 4x4 intra prediction and 4x4 residual coding... Cb Cr x2 DC AC Chroma 4x4 block order for 4x4 residual coding, shown as 16-25, and intra 4x4 prediction, shown as and Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 12

13 H.264/AVC Coder Input Video Signal Coder Control Control Data Split into Macroblocks 16x16 pixels - Transform/ Scal./Quant. Intra-frame Estimation Scaling & Inv. Transform Quant. Transf. coeffs Intra Prediction Data Entropy Coding Intra/Inter MB select Intra-frame Prediction Motion Compensation Motion Estimation Deblocking Filter Motion Data Output Video Signal Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 13

14 Common Elements with other Standards Macroblocks: 16x16 luma + 2 x 8x8 chroma samples Input: Association of luma and chroma and conventional sub-sampling of chroma (4:2:0) Block-wise motion compensation Motion vectors over picture boundaries Variable block-size motion Block transforms Scalar quantization I, P, and B coding types Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 14

15 H.264 Motion Compensation Accuracy Input Video Signal Split into Macroblocks 16x16 pixels - Decoder Coder Control Transform/ Scal./Quant. Scaling & Inv. Transform Control Data Quant. Transf. coeffs Entropy Coding Intra/Inter Intra-frame Prediction Motion- Compensation Motion Estimation De-blocking 16x16 16x8 8x16 8x8 Filter MB Types Output1 2 3 Video 8x8 Signal 8x4 4x8 4x4 8x Types Motion Data Motion vector accuracy 1/4 (6-tap filter) Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 15

16 H.264 Multiple Reference Frames Input Video Signal Split into Macroblocks 16x16 pixels - Decoder Coder Control Transform/ Scal./Quant. Scaling & Inv. Transform Control Data Quant. Transf. coeffs Entropy Coding Intra/Inter Intra-frame Prediction Motion- Compensation De-blocking Filter Output Video Signal Motion Estimation Multiple Reference Motion Frames Data Generalized B Frames Weighted Prediction Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 16

17 H.264 Intra Prediction Input Video Signal Split into Macroblocks 16x16 pixels - Decoder Intra/Inter Coder Control Transform/ Scal./Quant. Intra-frame Prediction Motion- Compensation Motion Estimation Scaling & Inv. Transform De-blocking Filter Directional spatial prediction (9 types for luma, 1 chroma) Control Data Q A B C D E F G H I a Quant. b c d J Transf. e f g coeffs h K i j k l L m n o p Output Video Signal Entropy Coding 0 e.g., Mode 3: diagonal down/right Motion prediction Data a, f, k, p are predicted by (A + 2Q + I + 2) >> 2 Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no

18 H.264 4x4 Transform Input Video Signal 4x4 Decoder Split into Block Integer Transform Macroblocks x16 pixels H = Intra/Inter Coder Control Transform/ Scal./Quant. Intra-frame Prediction Repeated transform of DC coeffs for 8x8 chroma and some 16x16 Intra luma blocks Motion- Compensation Scaling & Inv. Transform De-blocking Filter Control Data Quant. Transf. coeffs Output Video Signal Entropy Coding Motion Estimation Motion Data Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 18

19 Quantization of Transform Coefficients Scalar quantization Logarithmic step size control Smaller step size for chroma (per H.263 Annex T) Extended range of step sizes Can change to any step size at macroblock level Quantization reconstruction is one multiply, one add, one shift Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 19

20 Deblocking Filter Improves subjective quality and PSNR of the decoded picture Significantly superior to post filtering Filtering affects the edges of the 4x4 block structure Adaptive filtering removes blocking artifacts, but does not unnecessarily blur the visual content On slice level, the global filtering strength can be adjusted to the individual characteristics of the video sequence On edge level, filtering strength is made dependent on inter/intra, motion, and coded residuals On sample level, quantizer dependent thresholds can turn off filtering for every individual sample Specially strong filter for macroblocks with very flat characteristics almost removes tiling artifacts Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 20

21 Deblocking Filter q 0 q 1 q 2 One dimensional visualization of an edge position Filtering of p 0 and q 0 only takes place if: 1. p 0 -q 0 < α(qp) 2. p 1 -p 0 < β(qp) 3. q 1 -q 0 < β(qp) p 2 p 1 p 0 4x4 Block Edge Where β(qp) is considerably smaller than α(qp) Filtering of p 1 or q 1 takes place if additionally : 1. p 2 -p 0 < β(qp) or q 2 -q 0 < β(qp) (QP = quantization parameter) Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 21

22 Deblocking: Subjective Result for Intra Highly compressed first decoded intra picture at 0.28 bit/sample Without Filter With H264/AVC Deblocking Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 22

23 Deblocking: Subjective Result for Inter Highly compressed decoded inter picture Without Filter With H264/AVC Deblocking Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 23

24 Entropy coding Input Video Signal Split into Macroblocks 16x16 pixels - Decoder Coder Control Transform/ Scal./Quant. Inv. Scal. & Transform Control Data Quant. Transf. coeffs Entropy Coding Intra/Inter Intra-frame Prediction Motion- Compensation De-blocking Filter Output Video Signal Motion Estimation Motion Data Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 24

25 Variable length coding Exp-Golomb code for almost all symbols except for transform coefficients Context adaptive VLCs for coding of transform coefficients Number of coefficients is decoded Special treatment of values +1 and -1 Contexts are built dependent on transform coefficients Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 25

26 Context-Adaptive Arithmetic Coding (CABAC) update probability estimation Context modeling Binarization Probability estimation Coding engine Chooses a model conditioned on past observations Maps non-binary symbols to a binary sequence Adaptive binary arithmetic coder Uses the provided model for the actual encoding and updates the model Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 26

27 S Pictures General description Allows identical reconstruction of frames even when different reference frames are being used SP pictures use of motion-compensated prediction SI pictures can exactly approximate SP pictures Applications Bitstream switching or splicing Random access Fast-forward, fast-backward Error recovery and/or resiliency Resynchronization such as in Video Redundancy Coding Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 27

28 SP and SI Pictures l rec Quant. Transf. coeffs Scaling + Quantization Transform Scaling & Inv. Transform Entropy Decoding Control Data l pred Intra-frame Prediction De-blocking Filter Intra/Inter Motion- Compensation Output Video Signal Motion Data Motion Estimation Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 28

29 Comparison of H.264 to MPEG-4 MPEG-4: Advanced Simple Profile (ASP) Motion Compensation: 1/4 pel Global Motion Compensation H.264: Motion Compensation: 1/4 pel Using CABAC entropy coding 5 reference frames (News: 17) Both Sequence structure IBBPBBP... QP B =QP P +2 (step size: +25%) Search range: 32x32 around 16x16 predictor Lagrangian D+λR coder control [source: ITU-T VCEG] Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 29

30 RD Curves: Foreman (QCIF, 10Hz) Average PSNR(Y) [db] >30% MPEG-4 H.26L Bit-rate [kbit/s] [source: ITU-T VCEG] Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 30

31 Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 31

32 Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 32

33 Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 33

34 Performance Streaming Application Average bit-rate savings relative to: Coder MPEG-4 ASP H.263 HLP MPEG-2 H.264/AVC MP 37.44% 47.58% 63.57% MPEG-4 ASP % 42.95% H.263 HLP % [Wiegand, et al. 2003] Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 34

35 Example Streaming Test Result Y-PSNR [db] Tempete CIF 15Hz Bit-rate [kbit/s] MPEG-2 H.263 HLP MPEG-4 ASP H.264/AVC MP Test Points [Wiegand, et al. 2003] Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 35

36 Example Streaming Test Result 80% Tempete CIF 15Hz Rate saving relative to MPEG-2 70% 60% 50% 40% 30% 20% 10% 0% H.264/AVC MP MPEG-4 ASP H.263 HLP Y-PSNR [db] [Wiegand, et al. 2003] Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 36

37 Test Results for Real-Time Conversation Average bit-rate savings relative to: Coder H.263 CHC MPEG-4 SP H.263 Base H.264/AVC BP 27.69% 29.37% 40.59% H.263 CHC % 17.63% MPEG-4 SP % [Wiegand, et al. 2003] Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 37

38 Example Real-Time Conversation Result Y-PSNR [db] Paris CIF 15Hz Bit-rate [kbit/s] H.263-Base H.263 CHC MPEG-4 SP H.264/AVC BP Test Points [Wiegand, et al. 2003] Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 38

39 Example Real-Time Test Result 50% Paris CIF 15Hz Rate saving relative to H.263-Baseline 40% 30% 20% 10% H.264/AVC BP H.263 CHC MPEG-4 SP 0% Y-PSNR [db] [Wiegand, et al. 2003] Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 39

40 Test Results Entertainment-Quality Applications Average bit-rate savings relative to: Coder MPEG-2 H.264/AVC MP 45% [Wiegand, et al. 2003] Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 40

41 Example Entertainment-Quality Applications Result Y-PSNR [db] Entertainment SD (720x576i) 25Hz MPEG-2 H.264/AVC MP Bit-rate [Mbit/s] [Wiegand, et al. 2003] Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 41

42 Example Entertainment-Quality Applications Result 60% Entertainment SD (720x576i) 25Hz Rate saving relative to MPEG-2 50% 40% 30% 20% 10% 0% H.264/AVC MP Y-PSNR [db] [Wiegand, et al. 2003] Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 42

43 Further reading IEEE Transactions on Circuits and Systems for Video Technology, Special Issue on the H.264/JVC Video Coding Standard, July Bernd Girod: EE398B Image Communication II Video Coding Standards: H.264/AVC no. 43