Fine Granular Scalability -

Transcription

1 Fine Granular Scalability - A new framework for real-time streaming of video over the Internet Mihaela van der Schaar

2 Overview Why a new coding paradigm for Internet Video? Fine-Granular-Scalability (FGS) framework FGS features How does FGS compare with other coding schemes? New extensions FGS applications beyond video streaming Conclusions

3 Why a new coding solution? Internet characteristics! Wide range of available bandwidths & packet-loss rates => No QoS guarantees

4 Bandwidth variations 4.00 Bandwidth probability density New York - L.A Percent kbits/s

5 Bandwidth variations

6 Bandwidth variations Broadband Internet access has wider variation: Cable modem: from < 100 to > 1000 Kbit/sec DSL: from < 600 to > 6000 Kbit/sec

7 Packet-losses Time Packet Loss Ratio New York - L.A. 19:48 23:48 3:49 7:31 Percent 10:50 14:27 17:59 22:32 2:33 6:34 10:00 13:35

8 Why a new coding paradigm? Networks with time-varying bandwidth characteristics (e.g. Internet): No guarantees of QoS - wide range of available bandwidths and data loss rate Therefore required Easy adaptability to changing bandwidth Resilience to packet-losses Solution: scalable coding

9 Scalable Video in current standards Enhancement Layer(s) P B B B B R Enh2 P B B B B R Enh1 I P P P P R BL Base Layer

10 Scalable Video in current standards Alternative solution to simulcast Operate at a discrete set of bit-rates Supported bit-rates determined at Encoding-Time Overhead increases with the number of layers Not applicable for Packet Networks

11 What is important for a good solution? An adaptive solution for Internet video Easy bandwidth adaptability Encoding & transmission processes should be separated (transmission bit-rates do not need to be specified at encoding-time) Resilience to packet-losses (Inter-picture predictions within EL complicates loss recovery) Low server complexity -> unicast Support both unicast & multicast Low/scalable decoder complexity Solution: Fine-Granularity-Scalability

12 Fine-Granularity Scalability (FGS) R Max R min

13 Fine-Granularity Scalability (FGS) A Single Enhancement Layer R Max I B P B P R BL < R min Base Layer (e.g. MPEG-4 non-scalable codec)

14 Internet Video Streaming with FGS - Server side FGS Enhancement Layer R(t) FGS ELportion transmitted in real-time I B P B P Base Layer

15 Internet Video Streaming with FGS - Decoder side FGS Enhancement Layer I B P B P Base Layer

16 Advantages of the FGS structure Fine granular scalable enhancement layer Encoding & Transmission processes - separated Easy rate-control on multiple-streams for VOD Resilience to packet-losses Efficient for both unicast & multicast

17 FGS features Adaptive quantization Frequency weighting Hybrid temporal-snr scalability Error-resilience markers for wireless apps

18 Block-diagram of FGS-scalability Original Video Σ DCT Σ Q IDCT VLC IQ M U X BL stream Base-layer (BL) encoder MC Memory ME BL MVs

19 Block-diagram of FGS-scalability FGS Bit-plane DCT scanning and entropy coding SNR FGS stream FGS EL Encoder Σ Original Video Σ DCT Σ Q IDCT VLC IQ M U X BL stream Base-layer (BL) encoder MC Memory ME BL MVs

20 FGS bitplane coding BP(N) = lsb BP( 1) = msb DCT bitplanes

21 FGS AQ - Selective enhancement Macroblock k shifted up by one bitplane BP(N+1) BP(1) DCT bitplanes

22 How to select visually important regions? Possible solution: use real-time face-detection/tracking algorithms

23 FGS - Selective enhancement result

24 Hybrid FGS temporal-snr scalability - Motivation For optimal quality: trade-offs between individual image quality (SNR) and temporal resolution (higher framerates) Transmission frame-rate ~ transmission bit-rate => Hybrid SNR-temporal FGS scalability

25 Improved scalability A single FGS enhancement-layer Total flexibility in supporting SNR scalability with same frame-rate temporal scalability by increasing ONLY the frame-rate both FGS & temporal scalability Low added complexity

26 An all FGS temporal-snr scalability FGS Enhancement Layer R Max SNR SNR SNR SNR I P P P Base Layer R BL

27 An all FGS temporal-snr scalability FGS Enhancement Layer R Max SNR TEMP SNR TEMP SNR TEMP SNR TEMP I P P P Base Layer R BL

28 Real-time trade-off SNR-temporal Portion of the enhancement layer transmitted in real-time FGS Enhancement Layer I P P P Base Layer

29 Real-time trade-off SNR-temporal Portion of the enhancement layer transmitted in real-time FGS Enhancement Layer I P P P Base Layer

30 Real-time trade-off SNR-temporal Portion of the enhancement layer transmitted in real-time FGS Enhancement Layer I P P P Base Layer

31 Rate-control SNR/Temporal FGS Enhancement Layer Scenario A I P P Base Layer FGS Enhancement Layer Scenario B I P P Base Layer

32 Rate-control SNR/Temporal SNR versus temporal trade-off based on sequence characteristics (static): motion-info (e.g. MVs size) texture characteristics (e.g. Xi computed by base-layer RC) However, also dependent on Rt

33 Performance of FGS and FGS+FGST at same bit-rate 40 PSNR (db) FOREMAN at 500 kbit/s Scenario A Scenario B Frame number

34 SNR versus motion-smoothness PSNR (db) Frame number Scenario chosen: FOREMAN at 500 kbit/s B A B A

35 Internet video streaming - Encoding Video source R min, R max calculation Variablebandwidth network

36 Internet video streaming - Encoding Video source BL Encoder Base-layer stream of rate R BL R BL R min, R max calculation Variablebandwidth network

37 Internet video streaming - Encoding Hybrid Temporal/SNR FGS Video Encoder Enhancement-layer (R max R BL ) MC residual computation (Temporal-frames) SNR residual computation (SNR-frames) FGS Encoder FGS RC Video source BL Encoder Base-layer stream of rate R BL R BL R max R min, R max calculation Variablebandwidth network

38 Internet video streaming - Transmission Hybrid Temporal/SNR FGS Video Encoder Enhancement-layer (R max R BL ) MC residual computation (Temporal-frames) SNR residual computation (SNR-frames) FGS Encoder FGS RC Real-time scalable video rate controller User Input (remote/local) Video source BL Encoder R BL R max R min, R max, R calculation R EL stream (R R BL ) Variablebandwidth network

39 How does FGS compare with other streaming solutions? - Quality in absence of packet-losses - Quality in presence of packet-losses

40 Performance of FGS vs. SNR-scalability Multi-Level SNR Two-Level SNR FGS PSNR Bit Rate (kbit/s) No coding penalty due to the fine-granularity!

41 Single-layer switching (SLS) R max I P P P P R min +R 2 I P P P P R min +R 1 I P P P P R min

42 Performance of MPEG-4 FGS vs.sls under lossless conditions Foreman-CIF-10Hz-2secGOP PSNR (db) MPEG-4 SLS (10) MPEG-4 FGS 35 MPEG-4 SLS (3) Bit-rate (kbit/s)

43 FGS vs. SL FLYING - FGS with different base-layers (Base-layer bit rates around 200, 300, 500, 1000 kbit/sec) PSNR Bitrate (kbit/sec) BL1+ FGS Base layers (Non-scalable)

44 FGS vs. SL STEFAN - FGS with different base-layers (Base-layer bit rates around 200, 300, 500, 1000 kbit/sec) PSNR Bitrate (kbit/sec) BL1+ FGS Base layers (Non-scalable)

45 FGS vs. SL STEFAN - FGS with different base-layers (Base-layer bit rates around 200, 300, 500, 1000 kbit/sec) PSNR Bitrate (kbit/sec) BL1+ FGS BL2+FGS BL3+FGS Base layers (Non-scalable)

46 How to eliminate FGS coding penalty?

47 SL vs. FGS snr bit rate Base layers BL1+ FGS

48 SL vs. FGS snr bit rate Base layers BL1+ FGS

49 SL vs. FGS snr bit rate Base layers BL1+ FGS

50 MC-FGS structures Proposed in MPEG-4: One/Two-loop MC-FGS structures Largest gain can be obtained by 1-loop MC: Quality improvement (sequence dependent) - 2dB!!

51 Original FGS scheme Fine granular scalable Enhancement Layer I B P B P Prediction-based video Base Layer

52 Single-loop MC-FGS for all frames Prediction-based Fine granular scalable Enhanc ement Layer I B P B P Prediction-based video Base Layer

53 Single-loop MC-FGS for B-frames Prediction-based Fine granular scalable Enhanc ement Layer I B P B P Prediction-based video Base Layer

54 Why MC only for B-frames? B-frames represent 66% of FGS-stream in IBBP-GOPs B-frames have more accurate prediction No drift No propagation of losses

55 Single-loop MC-FGS for B-frames: NO DRIFT & NO Error Propagation Prediction-based Fine granular scalable Enhanc ement Layer I B P B P Prediction-based video Base Layer

56 Results - 1 MC-FGS loop - A PSNR difference 1MC-loop FGS all frames - Original FGS PSNR-difference (db) Coastguard Foreman Mobile bit-rate (kbit/s)

57 Results - 1 MC-FGS loop - B PSNR difference 1MC-loop FGS B-frames - Original FGS PSNR-difference (db) bit-rate (kbit/s) Coastguard Foreman Mobile

58 FGS resilience to packet-losses

59 Packet-loss in FGS FGS Enhancement Layer I B P B P Base Layer

60 FGS - packet-loss resilience properties Loss within an EL picture does not propagate Uneven packet-loss protection Lower packet-loss probability for FGS base-layer than for single-layer codec at same bit-rate Better FGS performance compared to conventional SNR scalability Error resilience over a wide range of bit-rates and packet-loss rates

61 Packet-loss in SLS R max I B P B P R min +R 2 I B P B P R min +R 1 I B P B P R min

62 Packet-loss in SLS R max I B P B P R min +R 2 I B P B P R min +R 1 I B P B P R min

63 Effective Packet-loss ratio (EP) Packet-Loss Ratio (PLR) 1% - 20 % Effective Packet-loss ratio EP=PLR.(1-RR) Recovery Ratio (RR) 0% - 100% EP ~ amount of unrecoverable packets Example: PLR=10%, RR=90% -> EP=1%

64 Packet-loss protection and concealment SLS Strategy A: loss detected => frame freeze Strategy B: employ MPEG-4 error resilience tools Protection levels: GOP / VOP/ Video-packets Concealment: copy data from previous frame

65 Packet-loss protection and concealment FGS: FGS base-layer: same as for SLS Packet-loss in EL => discard remaining (less significant) bit-planes No error concealment in FGS enhancement-layer

66 Packet-loss resilience (Rt=500kbit/s, EP=5%) SLS FGS

67 Packet-loss protection strategies for FGS Within the enhancement-layer: Fine Grained Loss Protection (UPP within enhancement-layer) R T EP n EP 2 EP 1 R BL EP BL

68 FGS packet-loss robustness Packet-loss robustness over various bit-rates and EP EPP: FGS outperforms SLS at moderate-high EP (5-10%) UPP between base/enhancement-layers and FGLP within enhancement-layer can provide significant resilience compared with SLS

69 Fine-Granular-Scalability (FGS) in MPEG-4 July 98 Activity initiated by MPEG-4 approved an FGS core-experiment Dec 98 FGS Requirements formally established. July 00 FGS reaches FPDAM-status. March 01 FGS will become an IS (MPEG-4 v4)

70 Preliminary conclusions MPEG-4 FGS solves the bandwidth-variation problem over the Internet A single enhancement-layer stream Totally flexible, efficient, and simple solution For both unicast and multicast Packet loss resilient Open standard Hybrid FGS temporal-snr scalability - important MPEG-4 tool for Internet Video

71 FGS - beyond video streaming Storage applications Memory

72 FGS - beyond video streaming Storage applications EL BL

73 FGS - beyond video streaming Storage applications

74 FGS - beyond video streaming Storage applications Disk full- What now?

75 FGS - beyond video streaming Storage applications

76 FGS - beyond video streaming Storage applications Progressive transmission of video data (caching) Joint bit-rate control for transmission etc. etc.

77 Acknowledgements Hayder Radha (Michigan State University, ) Yingwei Chen ( )

78 More info on FGS & internships: