Image and Video Coding Dr.-Ing. Henryk Richter Institute of Communications Engineering Phone: +49 381 498 7303, Room: W 8220 EMail: henryk.richter@uni-rostock.de
Literature / References l Gonzalez, R.; Woods, E. : Digital Image Processing, Prentice Hall 2008 l Jähne, B.: Digitale Bildverarbeitung, 5. Auflage, Springer-Verlag, 2002 l Tönnies, K. D.: Grundlagen der Bildverarbeitung, Pearson Studium, 2005 l Ohm, J.-R.: Digitale Bildcodierung, Springer-Verlag, 1995 l Rao K.R.: Techniques & Standards for Image, Video & Audio Coding, Prentice Hall 1996 l Ohm, J.-R.: Multimedia Communications Technology, Springer-Verlag 2004 l Wang, Y. et al.: Video Processing and Communications, Prentice Hall 2002 l Rao K.R. et al.: The transform and data compression handbook, CRC Press 2001 l Watkinson: MPEG-2, Focal Press 1999 l Pennebaker, W.B. et al.: JPEG still image compression standard, NY 1993 l Mitchell J. L. et al.: MPEG Video Compression Standard. Chapman and Hall 1997 l Taubman, D.S. et al.: JPEG2000, Kluwer Academics Publishers, 2002 l Richardson I.: H.264 and MPEG-4 Video Compression, Wiley & Sons 2003 l Gersho A. and Gray R. M.: Vector Quantization and Signal Compression, Kluwer Academics Publishers 1992 MAD 2015 2
Significant Conferences, Journals l Conferences l IEEE International Conference on Image Processing (ICIP) l Picture Coding Symposium (PCS) l International Conference on Visual Communications and Image Processing (VCIP) l Journals l IEEE Transactions on Circuits and Systems for Video Technology l IEEE Signal Processing Magazine l IEEE Communications Magazine l IEEE Transactions on Image Processing MAD 2015 3
General Transmission Model Sender source source encoding channel coding (error protection) modulation Noise, Fading, Interference Errors channel (transmission/storage) Receiver sink source decoding error correction demodulation source coding channel coding MAD 2015 4
Scope l Separation of source coding and channel coding l allows independent adaptation to - properties of information source and sink - properties of transmission channel l direct reusability of source coding output and transmission channel for other means l Drawbacks - joint source/channel coding enables balanced optimization between coding and error protection (graceful degradation) - unexploited redundancy by separate coding l Scope of image compression part of lecture: data compression = source coding MAD 2015 5
QCIF 176x144 CIF 352x288 DVD/DVB (4:3 format) NTSC 720x480@30 Hz PAL 720x576@25 Hz HDTV 720p 1280x720@60 Hz ITU-R BT.709 (16:9 format) HDTV 1080p 1920x1080@ 30 Hz UHDTV 2160p 3840x2160@120 Hz (~4k) CIF = common interchange format (5-30 Hz) QCIF = quarter common interchange format (10-30 Hz) MAD 2015 6
Necessity of data compression Width Height RGB FPS Data rate 720 576 3 25 = 31.1 MB/s 1920 1080 3 25 = 155.5 MB/s Capacity CD 700 MB DVD 8.5 GB Blu-ray 50 GB HDD 4 TB Storage time 22 s 4.5 s 4.5 min 54 s 26 min 5 min 35 h 7.1 h Bandwidth UMTS 1.4 MBit/s ADSL2+ 16 MBit/s VDSL 50 MBit/s Ethernet 1GE Transmission 90 min film 11 d 55 d 1 d 4.8 d 7.4 h 1.5 d 0.37 h 1.8 h MAD 2015 7
Aspect ratio and non square pixels Cinematic recordings traditionally recorded on 35 mm film horizontal image contraction on film (anamorphic representation) typical frame aspect ratios 2,35:1 (21:9, Cinemascope) 1,85:1 (100:54) 2,35 : 1 1,85 : 1 1,33 : 1 (4:3) Application to non-widescreen standards e.g. DVD 720x576 pixels (1,25:1) Letterbox Cropping (Pan/Scan) Anamorphic Representation of digital video commonly requires appropriate resampling MAD 2015 8
General data compression tools data compression decorrelation data reduction coding reversible concentration of signal energy reduction of symbol count increased information per symbol irreversible removal of irrelevancy reversible removal of redundancy optimization / adaptation MAD 2015 9
General lossy codec input decorrelation prediction transformation Encoder data reduction quantization coding precoding entropy coding transmission / storage output inverse transformation prediction Decoder data reconstruction inverse quantization decoding MAD 2015 10
General data compression tools data compression decorrelation data reduction coding reversible concentration of signal energy reduction of symbol count increased information per symbol irreversible removal of irrelevancy reversible removal of redundancy WS 2013 11
Characteristic Features l Removal or reduction of irrelevancy l Pre-assumption: l relevant and irrelevant parts are separated from each other as far as possible l Quantization: l number of symbols or samples keeps constant l precision decreases l reduced symbol set results in lower entropy l Sub-sampling: l l precision keeps constant number of symbols or samples decreases WS 2013 12
Example: spatial resolution 128x112 pixels 256x224 pixels 64x56 pixels 32x28 pixels 16x14 pixels WS 2013 13
Bi-cubic Original Windowed Sinc (Lanczos) Nearest Neighbor Subsampling Example: (reduction to 30% plus enlargement for display) WS 2013 14
The image matrix W Columns 0 1...... W-2 W-1 H Rows H-1 H-2...... 1 0 Pixel at position x,y s[x,y] Address in linear frame buffer y W + x MAD 2015 15
Image Matrix addressing and level scale Column n 1 Column Pitch Pixel White 255 Row Light Gray Row Pitch Medium Gray 128 n 2 Dark Gray Gray-level image (2D signal) Black 0 S = s [n 1,n 2 ] with s [n 1,n 2 ]=g 2 G = {0, 1,...,255} for 8 Bit per pixel l row-column addressing natural for most framebuffer types but beware of Matlab, where coordinates are swapped in the default toolsets and run from 1 (matrix/mathematics) notation instead of 0 (absolute offsets) MAD 2015 16
Organization of color components planar RGB packed RGB 24 Bit RGB as 3D matrix (logical) packed YCbCr 4:2:2, UYVY (Abekas) packed YCbCr 4:2:2, YVYU (Philips) packed RGB 32 Bit (RGBA) only a few common examples shown, RAW data organized to the needs of actual applications beware of Endianess issues when dealing with data accesses >8 Bit MAD 2015 17
General data compression tools data reduction subsampling quantization reduction of spatial resolution reduction of temporal resolution reduction of signal amplitude precision vector quantization scalar quantization mapping of similar vectors to a common representative vector mapping of similar values/amplitudes to a common representative value WS 2013 18